Project

General

Profile

Wiki » History » Version 109

Selvarani C, 27/02/2021 11:35

1 99 Selvarani C
{{toc}}
2 99 Selvarani C
3 100 Selvarani C
h1. %{color:BLUE}  Wiki%
4 2 Philippe May
5 48 Philippe May
h2. About the Auroville CSR Geomatics Studio
6 48 Philippe May
7 48 Philippe May
We are a team working on geographical and related information with an engineering and scientific approach. The concept of *geomatics* is explained here: https://en.wikipedia.org/wiki/Geomatics.
8 48 Philippe May
9 48 Philippe May
We develop, maintain and publish data on this web site: https://gis.auroville.org.in.
10 48 Philippe May
11 48 Philippe May
h3. Team
12 48 Philippe May
13 48 Philippe May
Currently, the team consists of:
14 48 Philippe May
15 48 Philippe May
* Bala
16 48 Philippe May
17 48 Philippe May
* Giulio
18 48 Philippe May
19 48 Philippe May
* Philippe
20 48 Philippe May
21 48 Philippe May
* Raj
22 48 Philippe May
23 48 Philippe May
* Ram
24 48 Philippe May
25 48 Philippe May
* Selvarani
26 48 Philippe May
27 48 Philippe May
h3. Collaborations
28 1 Philippe May
29 91 Selvarani C
Quick report of the collaborations the Geomatics Team the areas of work.
30 45 Philippe May
31 89 Selvarani C
32 88 Selvarani C
h1. Collaborations
33 88 Selvarani C
34 88 Selvarani C
Collaborations/coordination with other other groups, etc
35 88 Selvarani C
 
36 88 Selvarani C
|_.CSR Geomatics Description of Activity / Research / Project|_.AA|_.AVF|_.LB|_.FAMC|_.WG|_.CSR|_.Talam|_.DST|
37 88 Selvarani C
|Topographic Survey of Auroville City Area in coordination with L’avenir d’Auroville: Matrimandir entire compound and Residential Zone Sector 1 completed, Sector 2 half completed|yes|||||||yes|
38 88 Selvarani C
|Topographic Survey of specific projects area: Vibrance, Wasteless, Cultural Zone along the Crown Road, Baraka, Gardens of Unexpected |  |  |  |  |  |  |  |  |
39 88 Selvarani C
| Collective wastewater treatment systems health check-up survey: 68 plants evaluated | yes |  |  |  | yes | yes |  | yes |
40 88 Selvarani C
| Manual weekly monitoring of water level in selected wells on Auroville land on approximately 50 wells (number fluctuates depending on local conditions) | yes |  |  |  | yes |  |  | yes |
41 88 Selvarani C
| Collection of rainfall data through manual raingauges distributed to Aurovilians: data received regularly from at least 7 raingauges |  |  |  |  | yes |  |  | yes |
42 88 Selvarani C
| Collection of weather data through automatic weather station installed at CSR: data collected every minute, stored in the database, and published online in real time |  |  |  |  | yes | yes |  | yes |
43 88 Selvarani C
| Collaboration with Land Board for survey of identified land boundary stones: collection of coordinates of Government boundary stones for georefering of cadastral maps |  | yes | yes |  |  |  |  |  |
44 88 Selvarani C
| Collaboration with AV Foundation for compilation of land ownership map: geographic land records as provided by AV Foundation, protected by login access |  | yes |  |  |  |  |  |  |
45 88 Selvarani C
| Collaboration with L’avenir d’Auroville for data sharing and coordinated system set-up: organisation of geographic data for unique online platform to enhance planning processes (initial stage) | yes |  |  |  |  |  |  |  |
46 88 Selvarani C
| Collaboration with L’avenir d’Auroville and FAMC for database links: structural work on procedures for linking databases already existing with various agencies | yes |  |  | yes |  |  |  |  |
47 88 Selvarani C
| Study of application of drone topographic survey in Auroville context: pilot test conducted on Residential Zone with outside drone survey agency | yes |  |  |  |  | yes |  | yes |
48 88 Selvarani C
| Study of open source software for drone topographic survey: testing of softwares like OpenDroneMap, MeshLab, CloudeCompare to evaluate work flow, processes and limitations |  |  |  |  |  | yes |  | yes |
49 88 Selvarani C
| Surface water modelling (ongoing): research on evaluation of surface water collection potential through automatic processing of field data |  |  |  |  |  | yes |  | yes |
50 88 Selvarani C
| Collaboration with TALAM on research on radio-transmission for automatic wells level monitoring (ongoing): evaluation of application of radio-transmitted signals for water-related monitoring and metering (selected borewells, flow meters) |  |  |  |  |  |  | yes | yes |
51 88 Selvarani C
| Organisation of data collected in Auroville in the past (ongoing): standardisation of data in geospatial database |  |  |  |  |  | yes |  |  |
52 88 Selvarani C
| Provision for basic statistical analyses of geographic-related data (indicators/dashboard) |  |  |  |  |  | yes |  | yes |
53 88 Selvarani C
| Publication of collected data in standardised format on web portal gis.auroville.org.in |  |  |  |  |  | yes |  | yes |
54 88 Selvarani C
55 88 Selvarani C
* AA : L'avenir d'Auroville
56 88 Selvarani C
* AVF : Auroville Foundation
57 88 Selvarani C
* LB : Auroville Land Board
58 88 Selvarani C
* FAMC : Auroville Funds and Assets Management Committee
59 88 Selvarani C
* WG : Auroville Water Group
60 88 Selvarani C
* CSR : Auroville Centre for Scientific Research
61 88 Selvarani C
* TALAM : a project under CSR
62 88 Selvarani C
* DST : Department of Science and Technology, Ministry of Science and Technology, New Delhi
63 88 Selvarani C
64 5 Philippe May
h2. Our workflow
65 2 Philippe May
66 92 Selvarani C
67 92 Selvarani C
h2. Surveys
68 92 Selvarani C
69 92 Selvarani C
Most field surveys are executed by our team of surveyors, using advanced DGPS equipment.
70 92 Selvarani C
Other surveyors might also be contracted.
71 92 Selvarani C
72 92 Selvarani C
73 92 Selvarani C
h2. CAD
74 92 Selvarani C
75 92 Selvarani C
The survey data are imported to a CAD software (Aurocad/Civil 3D).
76 93 Selvarani C
77 93 Selvarani C
h1. Editing shapefiles in Autocad
78 93 Selvarani C
79 93 Selvarani C
h2. FEATURES CREATION
80 93 Selvarani C
81 93 Selvarani C
1.	Assign a CRS to the drawing (TM-AUSPOS) (MAPCSLIBRARY command)
82 93 Selvarani C
2.	Create features in CAD (Points, lines, polygons)
83 93 Selvarani C
3.	Export shapefile (a) from CAD (Output > DWG to SDF) (Convert to LL84 – 3D)
84 93 Selvarani C
85 93 Selvarani C
h2. FEATURES IMPORT INTO DB
86 93 Selvarani C
87 93 Selvarani C
4.	Create zip file of the shapefile
88 93 Selvarani C
5.	Upload into the GISAF Shapefiles Basket
89 93 Selvarani C
6.	Import the shapefile into DB
90 93 Selvarani C
7.	Delete the shapefile from Local Machine
91 93 Selvarani C
92 93 Selvarani C
93 93 Selvarani C
h2. FEATURES EDITING
94 93 Selvarani C
95 93 Selvarani C
8.	Open the table in QGis
96 93 Selvarani C
9.	Save as a shapefile (b) in TM AUSPOS CRS
97 93 Selvarani C
10.	In CAD, open a new drawing and assign ASUPOS CRS
98 93 Selvarani C
11.	Import the shapefile (b) (MapImport) with all Object Data
99 93 Selvarani C
12.	Edit features
100 93 Selvarani C
13.	Export shapefile (a) from CAD (Output > DWG to SDF) with ONLY the id selected (Data Tab > Select Attributes > Object Data > Filename > id) (Convert to LL84 – 3D)
101 93 Selvarani C
102 93 Selvarani C
103 93 Selvarani C
h2. FEATURES IMPORT INTO DB
104 93 Selvarani C
105 93 Selvarani C
14.	Create zip file of the shapefile
106 93 Selvarani C
15.	Upload into the GISAF Shapefiles Basket
107 93 Selvarani C
16.	Import the shapefile into DB
108 93 Selvarani C
17.	Delete the shapefile from Local Machine
109 92 Selvarani C
110 96 Selvarani C
 
111 92 Selvarani C
112 92 Selvarani C
h2. QGis
113 92 Selvarani C
114 92 Selvarani C
h3. Conventions
115 2 Philippe May
116 3 Philippe May
117 94 Selvarani C
h1. Shapefiles
118 94 Selvarani C
119 94 Selvarani C
We work with "QGis":https://en.wikipedia.org/wiki/QGIS , a widely used open source, free software for working on maps and geographical data.
120 94 Selvarani C
121 94 Selvarani C
"Shapefile":https://en.wikipedia.org/wiki/Shapefile is a standard file format for mapping, that Gisaf can import. QGis can open and save *shapefiles*.
122 94 Selvarani C
123 94 Selvarani C
We have defined some simple norms about these shapefiles for integration with Gisaf.
124 94 Selvarani C
125 94 Selvarani C
126 94 Selvarani C
127 94 Selvarani C
h2. Coordinate system
128 94 Selvarani C
129 94 Selvarani C
We use CRS SRID 32644.
130 94 Selvarani C
131 94 Selvarani C
132 94 Selvarani C
133 94 Selvarani C
h2. Column (attributes)
134 94 Selvarani C
135 94 Selvarani C
* All objects in a shapefile (layer) have a unique identifier named "id": numerical value.
136 94 Selvarani C
137 94 Selvarani C
138 94 Selvarani C
139 94 Selvarani C
h3. Field (attribute) names
140 94 Selvarani C
141 94 Selvarani C
* All fields are lower case (-UpperCase-, use: @lowercase@)
142 94 Selvarani C
143 94 Selvarani C
* They do not contain spaces, but underscores (-not this-, use: @but_that@)
144 94 Selvarani C
145 94 Selvarani C
* The field names cannot have more than 8 characters (-this_is_too_long-, use: @this_short@)
146 94 Selvarani C
147 94 Selvarani C
148 94 Selvarani C
h3. Attribute types
149 94 Selvarani C
150 94 Selvarani C
* The dates are written in ISO format: @YYYY-MM-DD@ (eg. @1968-02-25@)
151 94 Selvarani C
152 94 Selvarani C
h3. Fields to remove
153 94 Selvarani C
154 94 Selvarani C
* Eventually, remove the fields containing the coordinates: northing, easting, elevation, latitude, longitude, area, length, etc (these might be present when the data is imported from speadsheet)
155 94 Selvarani C
156 94 Selvarani C
h2. Foreign keys
157 94 Selvarani C
158 94 Selvarani C
We often deal with labels or categories of objects in a layer.
159 94 Selvarani C
160 94 Selvarani C
A common use case to explain: Alice creates a layer of stones, and wants to *tag* each stone with a rating: these are picked from a list of choices, like: _Beautiful_, _Interesting_, _Pretty_, etc.
161 94 Selvarani C
162 94 Selvarani C
For these kind of attribute: define a column like @rating_id@ (_something_ ending with @_id@) as a numerical value, and assign values 1, 2, 3, etc. The text is defined in another table (typically a CSV file), that looks like:
163 94 Selvarani C
164 94 Selvarani C
|_. id |_. name |
165 94 Selvarani C
| 1 | Beautiful |
166 94 Selvarani C
| 2 | Interesting |
167 94 Selvarani C
| 3 | Pretty |
168 94 Selvarani C
169 94 Selvarani C
h2. Code
170 94 Selvarani C
171 94 Selvarani C
We have defined a standard set of codes, that defines the type of data. They can be found here (TODO: add link).
172 94 Selvarani C
173 94 Selvarani C
Add a column @code_name@, matching with one the code, eg. @V25@ for TDEF.
174 94 Selvarani C
175 94 Selvarani C
h2. Surveyor
176 94 Selvarani C
177 94 Selvarani C
We keep a record of the people who realized the surveys (the _surveyors_).
178 94 Selvarani C
The shapefiles must contain an attribute @srvyr_id@, which refers to this table (TODO: add link).
179 94 Selvarani C
180 94 Selvarani C
h2. Accuracy
181 94 Selvarani C
182 94 Selvarani C
We keep a record of the accuracy of the surveys.
183 94 Selvarani C
The shapefiles must contain an attribute @accur_id@, which refers to this table (TODO: add link).
184 94 Selvarani C
185 94 Selvarani C
186 94 Selvarani C
h2. Date of survey
187 94 Selvarani C
188 94 Selvarani C
As nothing is absolutely permanent, it's also important to keep track of the date of the surveys: the shapefiles must contain an attribute @date@.
189 94 Selvarani C
190 94 Selvarani C
191 3 Philippe May
h2. Working with Gisaf
192 3 Philippe May
193 6 Philippe May
h3. Survey data
194 6 Philippe May
195 12 Philippe May
Raw survey data are contained in CSV files, typically downloaded from surveying instruments.
196 12 Philippe May
197 25 Philippe May
See more information of the process for the survey data (including a flow diagram): [[Survey data]]
198 12 Philippe May
199 97 Selvarani C
h1. Survey data
200 97 Selvarani C
201 97 Selvarani C
h2. Workflow summary
202 97 Selvarani C
203 98 Selvarani C
p=. !https://redmine.auroville.org.in/attachments/download/4792/Data_workflow.png!
204 97 Selvarani C
205 97 Selvarani C
h2. Import basket
206 97 Selvarani C
207 97 Selvarani C
Gisaf provides an "import basket" dedicated for raw survey data, which is generated by land survey equipment (Leica's Total Station and RTK). 
208 97 Selvarani C
209 97 Selvarani C
These are CSV files, like:
210 97 Selvarani C
<pre>
211 97 Selvarani C
100081,370633.969,1327742.157,51.187,,,
212 97 Selvarani C
100083,370628.876,1327702.913,51.565,T52,,
213 97 Selvarani C
100082,370628.729,1327720.019,51.261,T52,,
214 97 Selvarani C
100081,370633.969,1327742.154,51.179,,,
215 97 Selvarani C
100083,370628.876,1327702.913,51.565,T52,,
216 97 Selvarani C
20800,370633.969,1327742.154,51.180,,,
217 97 Selvarani C
20801,370618.795,1327713.172,52.817,E30,,
218 97 Selvarani C
20802,370623.674,1327711.436,51.283,B35,,
219 97 Selvarani C
20803,370619.314,1327713.407,51.383,B35,,
220 97 Selvarani C
</pre>
221 97 Selvarani C
222 97 Selvarani C
Each category (5th column) must be defined in the Category table (see [[Categories]]).
223 97 Selvarani C
224 97 Selvarani C
225 97 Selvarani C
h2. Organization of the raw survey data basket
226 97 Selvarani C
227 97 Selvarani C
The basket should be organized in a directory structure:
228 97 Selvarani C
229 97 Selvarani C
- Project name (these can be themselves put in a hierarchy of (sub)directories)
230 97 Selvarani C
231 97 Selvarani C
- Surveyor's organization
232 97 Selvarani C
233 97 Selvarani C
- Equipment (eg. TS, RTK)
234 97 Selvarani C
235 97 Selvarani C
- Survey files (eg. @Our_project-Some_comment-2018-02-23.txt@)
236 97 Selvarani C
237 97 Selvarani C
h3. Format of the survey file names
238 97 Selvarani C
239 97 Selvarani C
<pre>
240 97 Selvarani C
Our_project-Some_comment-2018-02-23.txt
241 97 Selvarani C
</pre>
242 97 Selvarani C
243 97 Selvarani C
The date of the survey follows the ISO date standard: @YYYY-MM-DD@.
244 97 Selvarani C
245 97 Selvarani C
246 97 Selvarani C
h2. Import to the database
247 97 Selvarani C
248 97 Selvarani C
When importing raw survey data files to the database, Gisaf does 2 steps as described below.
249 97 Selvarani C
It's worth noting that, in this process, no reprojection is done.
250 97 Selvarani C
251 97 Selvarani C
252 97 Selvarani C
h3. Feed the raw_survey table
253 97 Selvarani C
254 97 Selvarani C
Each point of the imported raw survey data file is inserted to the raw_survey table:
255 97 Selvarani C
256 97 Selvarani C
# Creation of a Point geometry: the raw_survey table has a geometry column for a single point (@geom@) with x,y and z coordinates
257 97 Selvarani C
# Save the @id@ of the original point *to the @orig_id@ column*
258 97 Selvarani C
# *A unique @id@ is computed* from the following fields: @id@, @project@, @equipment@, @date@
259 97 Selvarani C
# The project is saved in the @project_id@ column
260 97 Selvarani C
# The surveyor identification in @srvyr_id@
261 97 Selvarani C
# The date of survey is saved in the @date@ column
262 97 Selvarani C
# The accuracy is tagged in the @accur_id@, according to a mapping defined in the @accuracyequimentsurveyormapping@ table, which depends on the surveyor and equipment
263 97 Selvarani C
# The category of the point
264 97 Selvarani C
265 97 Selvarani C
266 97 Selvarani C
h3. Feed the @RAW_V_*@ tables
267 97 Selvarani C
268 97 Selvarani C
From the @raw_survey@ table, each point is then copied to its respective @RAW_V_@ table, with basically the same information.
269 97 Selvarani C
270 97 Selvarani C
These tables (which should be created manually or with the admin notebook called @create_tables@, as of today's writing), only contain points.
271 97 Selvarani C
272 97 Selvarani C
The project is saved along: see below.
273 97 Selvarani C
274 97 Selvarani C
275 97 Selvarani C
h2. Import the points
276 97 Selvarani C
277 97 Selvarani C
For categories that define points (opposite to lines and polygons, which require _line work_ carried over in CAD or in a GIS software, see [[Line work]]), the points can be imported automatically to their final destination: the @V_*@ tables.
278 97 Selvarani C
279 97 Selvarani C
Note: in this process, the geometries are reprojected.
280 97 Selvarani C
281 97 Selvarani C
282 97 Selvarani C
h3. Auto import of the points
283 97 Selvarani C
284 97 Selvarani C
The points found in the @RAW_V_*@ tables can be imported automatically, project per project, the project page of the admin interface.
285 97 Selvarani C
286 97 Selvarani C
287 97 Selvarani C
h3. Import of the line work (lines and polygons)
288 97 Selvarani C
289 97 Selvarani C
See [[Line work]]
290 97 Selvarani C
291 97 Selvarani C
The shapefiles generated manually (line work) should be put in the project's basket, and imported from it.
292 97 Selvarani C
293 12 Philippe May
h3. Categories
294 12 Philippe May
295 102 Selvarani C
The categories define the types of the geographical features and they are mapped according to ISO standard layer naming conventions: see https://www.nationalcadstandard.org/ncs5/pdfs/ncs5_clg_lnf.pdf
296 101 Selvarani C
297 101 Selvarani C
Gisaf uses:
298 101 Selvarani C
299 101 Selvarani C
* a table @category@ where the layers are defined
300 101 Selvarani C
* a table per category
301 101 Selvarani C
302 101 Selvarani C
h2. Fields for the categories
303 101 Selvarani C
304 101 Selvarani C
TODO
305 101 Selvarani C
306 101 Selvarani C
h2. Creation of the RAW_* tables
307 101 Selvarani C
308 101 Selvarani C
This step must be done manually (as of today's date of writing).
309 101 Selvarani C
310 6 Philippe May
311 1 Philippe May
h3. QGis: work on shapefiles
312 1 Philippe May
313 1 Philippe May
Go to [[shapefiles]].
314 5 Philippe May
315 5 Philippe May
316 103 Selvarani C
h1. Data analysis
317 18 Philippe May
318 103 Selvarani C
We use "Jupyter":https://jupyter.org , "Pandas":https://pandas.pydata.org/ and "GeoPandas":http://geopandas.org/ , accessible at http://gis.auroville.org.in/notebooks .
319 18 Philippe May
320 103 Selvarani C
For integration in the processes (execution of notebooks), there's "papermill":https://github.com/nteract/papermill . Systemd "timers":https://wiki.archlinux.org/index.php/Systemd/Timers are used to automatically schedule the notebooks on the server, ie. for the dashboards.
321 1 Philippe May
322 103 Selvarani C
There's a dedicated virtual machine for Jupyter, accessible from our local network at @jupyter.csr.av@.
323 103 Selvarani C
324 103 Selvarani C
h2. Organization of notebooks
325 103 Selvarani C
326 103 Selvarani C
The setup is organized in 2 parts, that are run with 2 instances of Jupyter for security reasons.
327 103 Selvarani C
328 103 Selvarani C
h3. Admin
329 103 Selvarani C
330 103 Selvarani C
The notebooks in the admin are mostly for maintenance: operations on the database, etc.
331 103 Selvarani C
332 103 Selvarani C
h3. Users
333 103 Selvarani C
334 103 Selvarani C
The notebooks are organized in folders, all under Gisaf's source code git repository, except the "Sandbox" one.
335 103 Selvarani C
336 103 Selvarani C
This notebook server connects to the database with a specific user (@jupyter@), which has been set on the database server with permissions to read all data (@readonly@) plus has write access to some tables dedicated to store analysis results.
337 103 Selvarani C
338 103 Selvarani C
h2. Integration with Gisaf
339 103 Selvarani C
340 103 Selvarani C
The notebook in @Templates@ demonstrates the usage of notebook in relation with Gisaf: mostly, how to use the @gisad.ipynb_tools@ module to access Gisaf models and the data from the database.
341 103 Selvarani C
342 103 Selvarani C
This module is part of gisaf: https://redmine.auroville.org.in/projects/gisaf/repository/revisions/master/entry/gisaf/ipynb_tools.py
343 103 Selvarani C
344 103 Selvarani C
h2. References
345 103 Selvarani C
346 103 Selvarani C
h3. Geopandas
347 103 Selvarani C
348 103 Selvarani C
Some nice examples of processing, using water shed and rain: https://geohackweek.github.io/vector/06-geopandas-advanced/
349 103 Selvarani C
350 103 Selvarani C
h3. Integration
351 103 Selvarani C
352 103 Selvarani C
A good example of how a company has integrated the same tools: https://medium.com/netflix-techblog/scheduling-notebooks-348e6c14cfd6
353 103 Selvarani C
354 103 Selvarani C
h2. Other docs
355 103 Selvarani C
356 103 Selvarani C
h3. From Swathi
357 5 Philippe May
358 5 Philippe May
h2. Hosting
359 5 Philippe May
360 5 Philippe May
The team is located in the CSR of Auroville.
361 5 Philippe May
362 104 Selvarani C
We have setup a server for hosting the software and database. CSR_server for technical information about the setup.
363 1 Philippe May
364 104 Selvarani C
h1. CSR server
365 1 Philippe May
366 104 Selvarani C
dream.csr.av (192.168.0.12)
367 104 Selvarani C
368 104 Selvarani C
- Debian 9
369 104 Selvarani C
- Xen hypervisor
370 104 Selvarani C
- libvirt for the orchestration of VMs
371 104 Selvarani C
- management with ansible
372 104 Selvarani C
373 104 Selvarani C
374 104 Selvarani C
h2. Dom0
375 104 Selvarani C
376 104 Selvarani C
h3. Installation
377 104 Selvarani C
378 104 Selvarani C
Found minor issues with the installation (eg. issues with HDDs, Dell EFI boot).
379 104 Selvarani C
This document starts from a working Debian Xen server.
380 104 Selvarani C
381 104 Selvarani C
-Installed on a mirror of 2*2TB drives with btrfs.-
382 104 Selvarani C
383 104 Selvarani C
Update (see #7156): re-installed the OS on /dev/sdc2, ext4, without RAID/LVM.
384 104 Selvarani C
385 104 Selvarani C
h3. Storage for domUs
386 104 Selvarani C
387 104 Selvarani C
A LVM dream.csr has been created for the domUs.
388 104 Selvarani C
389 104 Selvarani C
390 104 Selvarani C
h3. Networking
391 104 Selvarani C
392 104 Selvarani C
With systemd.networkd: bridge "br0" to the physical ethernet.
393 104 Selvarani C
394 104 Selvarani C
h3. Create a VM
395 104 Selvarani C
396 104 Selvarani C
Adjust the parameters from:
397 104 Selvarani C
398 104 Selvarani C
<pre>
399 104 Selvarani C
export vm_name=infra.csr.av
400 104 Selvarani C
export vm_ip_addr=172.16.0.3
401 104 Selvarani C
export vm_password=foobar
402 104 Selvarani C
</pre>
403 104 Selvarani C
404 104 Selvarani C
Create the domU:
405 104 Selvarani C
406 104 Selvarani C
<pre>
407 104 Selvarani C
root@dream:~# xen-create-image --broadcast=172.16.0.255 --dist=stretch --fs=ext4 --gateway=172.16.0.1 --hostname ${vm_name} --ip=${vm_ip_addr} --lvm=dream.csr --maxmem=512M --memory=256M --mirror=http://ftp.de.debian.org/debian/ --netmask=255.255.255.0 --password=${vm_password} --size=10G --swap=1G --vcpus=1 --bridge=br0
408 104 Selvarani C
</pre>
409 104 Selvarani C
410 104 Selvarani C
Note that IP address will be set in the VM, and the vm_ip_addr isn't actually used.
411 104 Selvarani C
412 104 Selvarani C
h2. DomUs
413 104 Selvarani C
414 104 Selvarani C
h3. Migrate XL to libvirt
415 104 Selvarani C
416 104 Selvarani C
After creation using xen-create-image, migrate the definition of the domU to libvirt:
417 104 Selvarani C
418 104 Selvarani C
<pre>
419 104 Selvarani C
virsh -c xen:/// domxml-from-native xen-xm /etc/xen/${vm_name}.cfg > /tmp/${vm_name}.xml
420 104 Selvarani C
virsh define /tmp/${vm_name}.xml
421 104 Selvarani C
</pre>
422 104 Selvarani C
423 104 Selvarani C
424 104 Selvarani C
From this point onward, one can log out from the dom0's console and use virsh or "Virtual Machine Manager" from your computer to administer the VM, eg:
425 104 Selvarani C
426 104 Selvarani C
* Set "Auto start" on dom0 boot
427 104 Selvarani C
* Set memory limits, etc
428 104 Selvarani C
429 104 Selvarani C
h3. Start the domU
430 104 Selvarani C
431 104 Selvarani C
Use "Virtual Machine Manager" or the command:
432 104 Selvarani C
433 104 Selvarani C
<pre>
434 104 Selvarani C
virsh -c xen+ssh://root@dream.csr.av/ start ${vm_name}
435 104 Selvarani C
</pre>
436 104 Selvarani C
437 104 Selvarani C
438 104 Selvarani C
h3. To do in a libvirt shell
439 104 Selvarani C
440 104 Selvarani C
Start a libvirt shell, with "Virtual Machine Manager" or with the command:
441 104 Selvarani C
442 104 Selvarani C
<pre>
443 104 Selvarani C
virsh -c xen+ssh://root@dream.csr.av/ console ${vm_name}
444 104 Selvarani C
</pre>
445 104 Selvarani C
446 104 Selvarani C
Log in as root in the libvirt console.
447 104 Selvarani C
448 104 Selvarani C
449 104 Selvarani C
h4. Network config
450 104 Selvarani C
451 104 Selvarani C
Add @/etc/systemd/network/50-wired.network@ (adjust the IP):
452 104 Selvarani C
453 104 Selvarani C
<pre>
454 104 Selvarani C
[Match]
455 104 Selvarani C
Name=eth0
456 104 Selvarani C
457 104 Selvarani C
[Network]
458 104 Selvarani C
Address=192.168.0.14/24
459 104 Selvarani C
Gateway=192.168.0.10
460 104 Selvarani C
DNS=192.168.0.10
461 104 Selvarani C
</pre>
462 104 Selvarani C
463 104 Selvarani C
Then:
464 104 Selvarani C
<pre>
465 104 Selvarani C
systemctl enable --now systemd-networkd.socket
466 104 Selvarani C
systemctl enable --now systemd-networkd.service
467 104 Selvarani C
systemctl disable --now networking.service
468 104 Selvarani C
</pre>
469 104 Selvarani C
470 104 Selvarani C
h4. Post-install ssh config
471 104 Selvarani C
472 104 Selvarani C
Allow ssh root login with password in @/etc/ssh/sshd_config@:
473 104 Selvarani C
474 104 Selvarani C
<pre>
475 104 Selvarani C
sed -i -e 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
476 104 Selvarani C
systemctl restart ssh.service
477 104 Selvarani C
</pre>
478 104 Selvarani C
479 104 Selvarani C
From this point onwards, one can close the console session opened via @libvirt@.
480 104 Selvarani C
481 104 Selvarani C
h3. DNS
482 104 Selvarani C
483 104 Selvarani C
Log in to the local DNS server with:
484 104 Selvarani C
485 104 Selvarani C
<pre>
486 104 Selvarani C
ssh root@infra.csr.av
487 104 Selvarani C
</pre>
488 104 Selvarani C
489 104 Selvarani C
Update the 2 files in @/etc/bind/zones@ (@zones/db.csr.av@ @zones/db.192.168.0@) and reload the DNS with:
490 104 Selvarani C
<pre>
491 104 Selvarani C
rndc reload
492 104 Selvarani C
</pre>
493 104 Selvarani C
494 104 Selvarani C
h3. DNS
495 104 Selvarani C
496 104 Selvarani C
Log in to the local DNS server, *update the 2 files in @/etc/bind9/zones@*, and run:
497 104 Selvarani C
<pre>
498 104 Selvarani C
rndc reload
499 104 Selvarani C
</pre>
500 104 Selvarani C
501 104 Selvarani C
h3. Allow yourself to log in easily with your ssh key
502 104 Selvarani C
503 104 Selvarani C
Copy your ssh key to the domU: run from your own computer:
504 104 Selvarani C
<pre>
505 104 Selvarani C
ssh-copy-id root@${ip_addr}
506 104 Selvarani C
</pre>
507 104 Selvarani C
508 104 Selvarani C
509 104 Selvarani C
h2. Ansible
510 104 Selvarani C
511 104 Selvarani C
Using the Ansible project developed in Blue Light: https://redmine.bluelightav.org/projects/ansible
512 104 Selvarani C
513 104 Selvarani C
h3. Prepare the host
514 104 Selvarani C
515 104 Selvarani C
Install Python
516 104 Selvarani C
517 104 Selvarani C
<pre>
518 104 Selvarani C
apt install -y python
519 104 Selvarani C
</pre>
520 104 Selvarani C
521 104 Selvarani C
522 104 Selvarani C
h2. Database
523 104 Selvarani C
524 104 Selvarani C
The Postgis database runs on its specific domU (gisdb.csr.av, 192.168.0.18).
525 104 Selvarani C
526 104 Selvarani C
h3. Installation
527 104 Selvarani C
528 104 Selvarani C
After installing the Postgis package (eg. assign the "postgis" Ansible's role), follow up to [[Db-support]]
529 104 Selvarani C
530 104 Selvarani C
531 104 Selvarani C
h2. Jupyter
532 104 Selvarani C
533 104 Selvarani C
The Jupyter server runs on its specific domU (jupyter.csr.av, 192.168.0.19).
534 104 Selvarani C
535 104 Selvarani C
h3. Installation
536 104 Selvarani C
537 104 Selvarani C
See #6990 .
538 105 Selvarani C
539 1 Philippe May
h3. Backup and restoration of the database
540 1 Philippe May
541 105 Selvarani C
h1. Database
542 105 Selvarani C
543 105 Selvarani C
h2. Troubleshooting
544 105 Selvarani C
545 105 Selvarani C
h3. Layers missing in the map's tree
546 105 Selvarani C
547 105 Selvarani C
Gisaf relies on counting features through Postgres statistics collector subsystem.
548 105 Selvarani C
549 105 Selvarani C
In case the server is restarted *dirty* (eg. without clean shutdown), then the count of the tables might be wrong or just 0, leaving the layers apparently empty and thus not even appearing.
550 105 Selvarani C
551 105 Selvarani C
The fix is as easy as:
552 105 Selvarani C
553 105 Selvarani C
<pre>
554 105 Selvarani C
sudo -u postgres psql avgis -c VACUUM
555 105 Selvarani C
</pre>
556 105 Selvarani C
557 105 Selvarani C
h2. Installation
558 105 Selvarani C
559 105 Selvarani C
This documentation assumes that the Postgis package has been installed (see [[CSR_server#Database]]).
560 105 Selvarani C
561 105 Selvarani C
h3. Configure the server
562 105 Selvarani C
563 105 Selvarani C
h4. Allow connections from other hosts in the local network
564 105 Selvarani C
565 105 Selvarani C
Set the server to listen to addresses, set listen_addresses to @*@ in @/etc/postgresql/9.6/main/postgresql.conf@.
566 105 Selvarani C
567 105 Selvarani C
568 105 Selvarani C
Allow the connections, add in @/etc/postgresql/9.6/main/pg_hba.conf@:
569 105 Selvarani C
570 105 Selvarani C
<pre>
571 105 Selvarani C
host all all 192.168.0.0/24 md5
572 105 Selvarani C
</pre>
573 105 Selvarani C
574 105 Selvarani C
575 105 Selvarani C
h2. Creation of the database
576 105 Selvarani C
577 105 Selvarani C
As @postgres@ user:
578 105 Selvarani C
579 105 Selvarani C
<pre>
580 105 Selvarani C
createdb -E utf8 -T template0 avgis
581 105 Selvarani C
</pre>
582 105 Selvarani C
583 105 Selvarani C
h2. Backups
584 105 Selvarani C
585 105 Selvarani C
h3. Primary
586 105 Selvarani C
587 105 Selvarani C
The database is backed up every day at midnight. The dump file is located in @/var/backups/postgres/@.
588 105 Selvarani C
589 105 Selvarani C
h3. Secondary
590 105 Selvarani C
591 105 Selvarani C
There are other backups (daily, weekly, monthly) thanks to Debian package @autopostgresqlbackup@), located (default) in @/var/lib/autopostgresqlbackup@.
592 105 Selvarani C
593 105 Selvarani C
h3. Tertiary (dom0)
594 105 Selvarani C
595 105 Selvarani C
The whole virtual machine is backed up by BackupNinja on the "dom0" controller, using:
596 105 Selvarani C
- rdiff backups every day
597 105 Selvarani C
- tar files on Saturdays.
598 105 Selvarani C
599 105 Selvarani C
See @/etc/backups.d@ on the dom0 (192.168.0.12).
600 105 Selvarani C
601 105 Selvarani C
h3. Remote
602 105 Selvarani C
603 105 Selvarani C
TODO: remote backup.
604 105 Selvarani C
605 105 Selvarani C
606 105 Selvarani C
h2. Restoration
607 105 Selvarani C
608 105 Selvarani C
If the VM is not shutdown properly, there's a chance that the database is corrupt, and needs to be restored from one of the backups.
609 105 Selvarani C
610 105 Selvarani C
After the restoration, restart gisaf:
611 105 Selvarani C
<pre>
612 105 Selvarani C
systemctl restart uwsgi.service
613 105 Selvarani C
</pre>
614 105 Selvarani C
615 105 Selvarani C
h3. From primary backup
616 105 Selvarani C
617 105 Selvarani C
*Note*: the roles aren't restored with this method.
618 105 Selvarani C
619 105 Selvarani C
With user @postgres@:
620 105 Selvarani C
<pre>
621 105 Selvarani C
# Optionally, rename the corrupt database (selecting a name for a database like "avgis_c2")...
622 105 Selvarani C
psql -c "ALTER DATABASE avgis RENAME TO avgis_c2;"
623 105 Selvarani C
# ... or drop the existing database
624 105 Selvarani C
psql -c "drop database avgis;"
625 105 Selvarani C
# Create a new database:
626 105 Selvarani C
createdb -E utf8 -T template0 avgis
627 105 Selvarani C
# Restore the database
628 105 Selvarani C
pg_restore -d avgis /var/backups/postgres/avgis.pg_dump
629 105 Selvarani C
</pre>
630 105 Selvarani C
631 105 Selvarani C
h3. From secondary backup
632 105 Selvarani C
633 105 Selvarani C
@autopostgresqlbackup@ backs up the roles in @postgres_globals@.
634 105 Selvarani C
635 105 Selvarani C
<pre>
636 105 Selvarani C
zcat /var/lib/autopostgresqlbackup/daily/postgres_globals/postgres_globals_2018-10-24_06h25m.Wednesday.sql.gz | psql
637 105 Selvarani C
zcat /var/lib/autopostgresqlbackup/daily/avgis/avgis_2018-10-24_06h25m.Wednesday.sql.gz | psql
638 105 Selvarani C
</pre>
639 17 Philippe May
640 17 Philippe May
h2. Gear
641 17 Philippe May
642 17 Philippe May
h3. Survey equipment
643 17 Philippe May
644 17 Philippe May
See [[survey equipment]]
645 17 Philippe May
646 17 Philippe May
h3. Weather station
647 17 Philippe May
648 17 Philippe May
See [[Ambient_Weather_weather_station]]
649 11 Philippe May
650 106 Selvarani C
h1. Ambient Weather weather station
651 106 Selvarani C
652 106 Selvarani C
We have purchased a WS2902A weather station (https://www.ambientweather.com/amws2902.html).
653 106 Selvarani C
Firmware version: 4.0.2.
654 106 Selvarani C
655 106 Selvarani C
h2. Manual
656 106 Selvarani C
657 106 Selvarani C
The operating manual of the weather station can be found at https://p10.secure.hostingprod.com/@site.ambientweatherstore.com/ssl/Manuals/WS-2902C.pdf
658 106 Selvarani C
659 106 Selvarani C
h2. Connection
660 106 Selvarani C
661 106 Selvarani C
h3. Wifi
662 106 Selvarani C
663 106 Selvarani C
Set up the wifi of the console using the "Ambient Tool" phone application. IP address given by DHCP on the router is: 192.168.1.101
664 106 Selvarani C
665 106 Selvarani C
h3. Local communication
666 106 Selvarani C
667 106 Selvarani C
Fail so far: the only exposed port is TCP/45000. Telnet doesn't show any activity. Nothing found on Internet on this protocol.
668 106 Selvarani C
669 106 Selvarani C
One interesting project may come, hijacking the connection to cloud services: https://www.wxforum.net/index.php?topic=35033.0
670 106 Selvarani C
671 106 Selvarani C
672 106 Selvarani C
h3. Cloud connection
673 106 Selvarani C
674 106 Selvarani C
We'll create an account on AmbientWeather.net (and eventually on WUnderground.net and/or weathercloud.net), and:
675 106 Selvarani C
676 106 Selvarani C
* have the console upload data to there
677 106 Selvarani C
* gisaf to retrieve our WS data from there
678 106 Selvarani C
679 106 Selvarani C
h1. Notes on Console
680 106 Selvarani C
681 106 Selvarani C
The daily rainfall data displayed in  the console resets at 00.30 every night
682 11 Philippe May
683 7 Philippe May
h2. Plan for future
684 7 Philippe May
685 8 Philippe May
Beside living well, see [[plan]].
686 8 Philippe May
687 107 Selvarani C
h1. Plan
688 107 Selvarani C
689 107 Selvarani C
Some interesting projects that might be integrated:
690 107 Selvarani C
691 107 Selvarani C
* https://github.com/Oslandia/albion : Build 3D geological model from wells information
692 107 Selvarani C
693 49 Philippe May
h2. Other
694 1 Philippe May
695 50 Philippe May
* [[GDAL (OGR) tools]]
696 49 Philippe May
697 108 Selvarani C
h1. GDAL (OGR) tools
698 108 Selvarani C
699 108 Selvarani C
"GDAL":https://gdal.org/ is a translator library for raster and vector geospatial data formats. It is used by many software (including QGIS and many other open source ones, including Gisaf). Some command line utilities are supplied, like:
700 108 Selvarani C
701 108 Selvarani C
* @ogr2ogr@ can easily convert one data format to another
702 108 Selvarani C
* @ogrinfo@ displays information about files.
703 108 Selvarani C
704 108 Selvarani C
h2. Using Windows
705 108 Selvarani C
706 108 Selvarani C
On a computer with Windows and GQIS installed:
707 108 Selvarani C
708 108 Selvarani C
1. Open a command line console (eg. <Windows Key> to display the Start menu, then just type @cmd@ and <Enter>)
709 108 Selvarani C
2. In the console window, type (adjust with the QGIS version and location, this seems to be the standard one):
710 108 Selvarani C
711 108 Selvarani C
<pre>
712 108 Selvarani C
"c:\Program Files\QGis 3.10\OSGeo4W.bat"
713 108 Selvarani C
</pre>
714 108 Selvarani C
715 108 Selvarani C
3. GDAL utilities can be used: @ogr2ogr@, etc.
716 108 Selvarani C
717 108 Selvarani C
h2. Example: convert Geopackage to Shapefiles
718 108 Selvarani C
719 108 Selvarani C
Output the content of the geopackage @9wdoogfr_2019-11-13_12_26_07.gpkg@ to the folder @shapefiles@:
720 108 Selvarani C
721 108 Selvarani C
<pre>
722 108 Selvarani C
ogr2ogr -progress -f "ESRI Shapefile" shapefiles 9wdoogfr_2019-11-13_12_26_07.gpkg
723 108 Selvarani C
</pre>
724 108 Selvarani C
725 108 Selvarani C
To output the content of the geopackage @9wdoogfr_2019-11-13_12_26_07.gpkg@ to the @root@ folder:
726 108 Selvarani C
727 108 Selvarani C
<pre>
728 108 Selvarani C
ogr2ogr -progress -f "ESRI Shapefile" c:\shapefiles 9wdoogfr_2019-11-13_12_26_07.gpkg
729 108 Selvarani C
</pre>
730 108 Selvarani C
731 108 Selvarani C
h2. With reprojection
732 108 Selvarani C
733 108 Selvarani C
Same as above, reprojecting to UTM44N: 
734 108 Selvarani C
735 108 Selvarani C
<pre>
736 108 Selvarani C
ogr2ogr -progress -f "ESRI Shapefile" -t_srs EPSG:32644 c:\shapefiles 9wdoogfr_2019-11-13_12_26_07.gpkg
737 108 Selvarani C
</pre>
738 108 Selvarani C
739 8 Philippe May
h2. Links and references
740 8 Philippe May
741 10 Philippe May
[[links]]
742 16 Philippe May
743 109 Selvarani C
h1. Links
744 109 Selvarani C
745 109 Selvarani C
h2. Water management
746 109 Selvarani C
747 109 Selvarani C
h3. Modflow
748 109 Selvarani C
749 109 Selvarani C
* https://water.usgs.gov/ogw/modflow/
750 109 Selvarani C
751 109 Selvarani C
The reference software for underground water modelling and simulation.
752 109 Selvarani C
In conjunction with flopy (https://water.usgs.gov/ogw/flopy/) and Jupyter (https://jupyter.org/), it provides a _relatively easy_ to use interface.
753 109 Selvarani C
754 109 Selvarani C
755 109 Selvarani C
h3. Freewat
756 109 Selvarani C
757 109 Selvarani C
* http://www.freewat.eu/project
758 109 Selvarani C
759 109 Selvarani C
This project is partly based on modflow, and integrates with QGis.
760 109 Selvarani C
761 109 Selvarani C
h2. QGIS
762 109 Selvarani C
763 109 Selvarani C
* https://www.qgis.org/en/site/
764 109 Selvarani C
765 109 Selvarani C
* Tools for Geology
766 109 Selvarani C
Construction of geological cross sections in QGIS - http://www.geokincern.com/?p=1452
767 109 Selvarani C
768 109 Selvarani C
769 109 Selvarani C
h2. Autocad
770 109 Selvarani C
771 109 Selvarani C
* Overview of Converting Geospatial Data to Drawing Objects: 
772 109 Selvarani C
http://docs.autodesk.com/CIV3D/2013/ENU/index.html?url=filesMAPC3D/GUID-C38FD485-3CC2-4B52-8264-0D8C0F45422B.htm,topicNumber=MAPC3Dd30e41809
773 109 Selvarani C
774 109 Selvarani C
* CAD-DB connection:
775 109 Selvarani C
https://knowledge.autodesk.com/support/autocad-civil-3d/learn-explore/caas/video/youtube/watch-v-AQoB--nyUJA.html
776 109 Selvarani C
777 109 Selvarani C
778 109 Selvarani C
h2. Orfeo
779 109 Selvarani C
780 109 Selvarani C
* https://www.orfeo-toolbox.org/
781 109 Selvarani C
782 109 Selvarani C
Remote sensing
783 16 Philippe May
784 16 Philippe May
h2. Old docs
785 16 Philippe May
786 15 Philippe May
[[Shapefiles]]
787 15 Philippe May
[[Data (measurements, auxiliary tables)]]
788 16 Philippe May
789 13 Philippe May
h2. Pavneet's docs (imported from gisaf's wiki)
790 1 Philippe May
791 15 Philippe May
792 14 Philippe May
[[Rules of Map making - What all Maps should have!]]
793 14 Philippe May
[[Survey Database]]
794 14 Philippe May
[[Field to finish]]
795 14 Philippe May
[[Survey Data Post-Processing]]
796 14 Philippe May
[[Wells Documentation]]
797 14 Philippe May
[[Civil 3D useful commands]]
798 14 Philippe May
[[Online references for Civil 3D]]
799 1 Philippe May
[[connections in QGIS- using browser panel and Add postGIS]]
800 33 Pavneet Kaur
[[Reconcilation of Raw survey data using pgAdmin]]
801 22 Pavneet Kaur
[[importing RAW data to GISAF]]
802 26 Pavneet Kaur
[[Editing Z value of features in Shapefiles in QGIS]]
803 34 Pavneet Kaur
[[Miscellaneous- Civil 3D]]
804 36 Pavneet Kaur
[[Documentation- Rain Gauge]]
805 40 Pavneet Kaur
[[Documentation- Wells Monitoring (Piezometer by Bala)]]
806 40 Pavneet Kaur
[[Documentation- Flow meter, by Bala]]
807 42 Pavneet Kaur
[[Documentation- DST- Vegetation Indexing]]
808 44 Pavneet Kaur
[[Documentation- DST- Interpolation]]
809 43 Pavneet Kaur
[[Documentation- DST- Survey- Office workflow]]
810 40 Pavneet Kaur
[[From CAD to GIS, by Giulio]]
811 41 Pavneet Kaur
[[QGIS- Miscellaneous]]
812 52 Giulio Di Anastasio
813 1 Philippe May
h2. Giulio's documentation
814 59 Giulio Di Anastasio
815 52 Giulio Di Anastasio
[[Documentation - Reconciliation of points using Gisaf]]
816 65 Giulio Di Anastasio
[[Documentation - Status and Status Changes]]
817 66 Giulio Di Anastasio
[[Documentation - Tags retained after re-import of same geometry]]
818 67 Giulio Di Anastasio
819 67 Giulio Di Anastasio
820 67 Giulio Di Anastasio
h1. Access to data
821 67 Giulio Di Anastasio
822 67 Giulio Di Anastasio
h2. Connection to server directly from CSR
823 67 Giulio Di Anastasio
824 67 Giulio Di Anastasio
To connect to the server directly without going through Aurinoco server, the correct url is
825 67 Giulio Di Anastasio
http://gis.csr.av
826 67 Giulio Di Anastasio
827 67 Giulio Di Anastasio
h2. Connection to Gisaf via QGis through WFS / OGC API
828 67 Giulio Di Anastasio
829 67 Giulio Di Anastasio
This works only on QGis from version 3.14.15 onward
830 67 Giulio Di Anastasio
831 67 Giulio Di Anastasio
In the browser, click on WFS/OGC API, then right-click to create a new connection
832 67 Giulio Di Anastasio
Give a name (e.g. OGC API Qgis Gisaf)
833 67 Giulio Di Anastasio
Give the url https://gis.auroville.org.in/ogcapi
834 67 Giulio Di Anastasio
835 67 Giulio Di Anastasio
Under the WFS Options box, on Version dropdown, the default option "Maximum" works just fine
836 67 Giulio Di Anastasio
Click on OK
837 67 Giulio Di Anastasio
The list of layers will appear in the Browser under WFS/OGC API.
838 68 Giulio Di Anastasio
839 68 Giulio Di Anastasio
840 68 Giulio Di Anastasio
h1. How to create a new projection in QGis
841 68 Giulio Di Anastasio
842 68 Giulio Di Anastasio
To create a new projection in QGis, go to menu "Settings", and click on "Custom Projections".
843 68 Giulio Di Anastasio
A pop-up window appears with a list of all projections defined in QGis projects used by the user so far.
844 68 Giulio Di Anastasio
Click on the green "+" sign on the right top part of the window to create a new projection.
845 68 Giulio Di Anastasio
In the "Name" box, type "TM CSRAUSPOS SF1" (which means TM = Transverse Mercator projection; CSRAUSPOS = theparameters for this projection are derived from the processing of DGPS raw data by AUSPOS -  Online GPS Processing Service - https://www.ga.gov.au/scientific-topics/positioning-navigation/geodesy/auspos; SF1 = Scale Factor is 1).
846 68 Giulio Di Anastasio
In the "Format" dropdown list, select "Proj String (legacy - Not Recommended)"
847 68 Giulio Di Anastasio
In the "Parameters" box, paste the following "+proj=tmerc +lat_0=12.01605433+lon_0=79.80998934 +k=1 +x_0=370455.630 +y_0=1328608.994 +ellps=WGS84+towgs84=0,0,0,0,0,0,0 +units=m +no_defs".
848 68 Giulio Di Anastasio
849 68 Giulio Di Anastasio
Finally, click on OK.
850 68 Giulio Di Anastasio
851 68 Giulio Di Anastasio
In a more explicit  way, the parameters mean the following:
852 68 Giulio Di Anastasio
Map Projection:        TransverseMercator (TM)
853 68 Giulio Di Anastasio
False Easting:          370455.6300
854 68 Giulio Di Anastasio
False Northing:        1328608.9940
855 68 Giulio Di Anastasio
Latitude of Origin:    12°00'57.79560" (DMS)   12.01605433 (DD)
856 68 Giulio Di Anastasio
Central Meridian:     79°48'35.96164"  (DMS)   79.80998934 (DD)
857 68 Giulio Di Anastasio
Scale Factor:             1.00000000
858 68 Giulio Di Anastasio
Zone Width:              6.0°
859 69 Giulio Di Anastasio
860 69 Giulio Di Anastasio
h1. Elimination of Duplicate points – General criteria
861 69 Giulio Di Anastasio
862 69 Giulio Di Anastasio
It might happen that the same physical feature (e.g. a tree, or a pole) is surveyed more than once: this can happen because there are many physical features in an area, and the survey needs more than one station. So, for example a tree is surveyed from a station, and gets a serial number on that date. When the station is then changed, it might happen that the same tree is resurveyed: another serial number is given, and possibly a different date, if the survey from the second station happened on a different day.
863 69 Giulio Di Anastasio
It is clear that the same tree is then represented with two different points, which means that two different trees exist: but only one tree really exist in the physical reality.
864 69 Giulio Di Anastasio
It is clear that one of the two points is redundant and needs to be removed. If this is noted by the surveyor directly in the field, then the issue is solved by the surveyor himself during processing time.
865 69 Giulio Di Anastasio
If instead, due to various reasons, it was not noted by the surveyor in the field, it will need to be cleaned after the processing, possibly by post-processing staff.
866 69 Giulio Di Anastasio
How to identify duplicate points?
867 69 Giulio Di Anastasio
The following criteria can be used:
868 69 Giulio Di Anastasio
1.	The distance between the two points is less than 30 cm (trees are surveyed if their trunk diameter is at least about 20 cm, so in 30 cm cannot exist two of them)
869 69 Giulio Di Anastasio
2.	The orig_id (serial number) of the points are not in series
870 69 Giulio Di Anastasio
3.	The survey date is not the same
871 69 Giulio Di Anastasio
4.	In case of trees, the species of trees is the same
872 69 Giulio Di Anastasio
5.	5. In case of trees, the tree type is not TDEF (because TDEF are mapped irrespective of their diameter, so they can actually have a small trunk, and two of them might exist in 30 cm), not OT (many TDEF species are surveyed as OT if not otherwise indicated by a botanist)
873 69 Giulio Di Anastasio
6.	The context needs to be evaluated: if one tree is deleted in an area where many trees exist in a limited space, then loosing one in the map is not a big error. If instead one tree is deleted where there are very few trees, then it might be a big loss.
874 70 Giulio Di Anastasio
875 70 Giulio Di Anastasio
876 70 Giulio Di Anastasio
h1. Linework for the Survey Area
877 70 Giulio Di Anastasio
878 70 Giulio Di Anastasio
h2. 1.	Creation of Initial Linework in QGIS using Survey points import - (Ram, System 4)
879 70 Giulio Di Anastasio
880 70 Giulio Di Anastasio
Initial Linework in QGIS is started by surveyor with the knowledge from the Field. For this step, points are simply imported into the QGIS from the field text file (.csv or .txt). CRS needs to be TM-AUSPOS. The box of “First record has field names” shall not be ticked. In Point Coordinates, select the correct field for x, for y and for z (usually “field_2” for x, “field_3” for y and “field_4” for z). Points can be styled using the “Categorized” style in “Symbology”, using “Field_5” as value, or using a Rule-based symbology using the category (field 5) as filter.
881 74 Selvarani C
Linework is created by connecting points having same description and belonging to the same physical feature. *All line and polygon features are created as lines*.
882 70 Giulio Di Anastasio
The Initial Linework for the Survey Area is also stored temporarily in 
883 70 Giulio Di Anastasio
884 74 Selvarani C
+D: > AVSM > Zone-Survey number (eg RZ-01) > Survey Area (eg J) > Temporary WD+
885 70 Giulio Di Anastasio
886 70 Giulio Di Anastasio
h2. Note: The line shapefiles / Geopackages shall be in CRS: TM AUSPOS
887 70 Giulio Di Anastasio
888 70 Giulio Di Anastasio
h2. 2.	Creation of final working drawing Shapefiles / Geopackages - (Selvarani, System 1)
889 1 Philippe May
890 1 Philippe May
Final working drawing Shapefiles / Geopackages are created from the Initial Linework of Survey Area.
891 70 Giulio Di Anastasio
As the Surveyor draws all features as lines (both for lines and polygons features), the following actions shall be done:
892 74 Selvarani C
1.	*If features are lines:*
893 70 Giulio Di Anastasio
•	Export the shapefile / geopackage into the final working drawing folder (Final WD), in separate folders according to its type (e.g. BLDG, FENC, ROAD, etc).
894 70 Giulio Di Anastasio
895 70 Giulio Di Anastasio
h2. The CRS for the export shall be EPSG:4326 - WGS 84
896 70 Giulio Di Anastasio
897 74 Selvarani C
2.	*If features are polygons:*
898 1 Philippe May
•	Lines shall be converted into polygons:
899 74 Selvarani C
900 1 Philippe May
to do it, first click on the layer to be converted to make it active (e.g. WD-CZ-01-F-LL84_V-BLDG-MHOL------E), then go to “Vector” Menu, click on Geometry Tools, click on Line to Polygons:
901 87 Selvarani C
902 84 Selvarani C
!https://redmine.auroville.org.in/attachments/download/9760/Line%20to%20Polygon%20Menu.png!
903 70 Giulio Di Anastasio
904 86 Selvarani C
905 1 Philippe May
The new window for “Lines to Polygons” conversion will appear:
906 86 Selvarani C
907 86 Selvarani C
908 83 Selvarani C
!https://redmine.auroville.org.in/attachments/download/9762/Lines%20to%20Polygon%20Window.png!
909 1 Philippe May
910 71 Giulio Di Anastasio
•	Always cross check the input layer, to make sure that the input layer is the active one
911 71 Giulio Di Anastasio
•	Save the output in a temporary layer
912 74 Selvarani C
•	The temporary layer will be listed in the list of layers, it shall be exported to the saving location as +D: > Survey > Zone-Survey Number > Final WD > Survey Area SHP+ (eg . D: > Survey > GB-01 > Final WD > A-Shp)
913 72 Giulio Di Anastasio
914 74 Selvarani C
h2. The CRS for the export shall be EPSG:4326 - WGS 84
915 74 Selvarani C
916 72 Giulio Di Anastasio
Once all the shapefiles / geopackages are exported in Final WD, for each of the newly exported layers the Topology Checker Tool shall be used.
917 1 Philippe May
918 74 Selvarani C
h2. Linework for the whole Survey Zone
919 74 Selvarani C
920 74 Selvarani C
h2. 1.	Merging Shapefiles / Geopackages - (Selvarani, System 1)
921 74 Selvarani C
922 72 Giulio Di Anastasio
A copy of the Zone Master shapefiles / geopackages are taken from System 4 and stored in Temp Folder on Desktop in System 1.
923 1 Philippe May
Master shapefiles / geopackages are merged with the Survey Area shapefiles / geopackages:
924 72 Giulio Di Anastasio
•	To do it, go to “Vector” Menu, click on Geoprocessing Tools, then click on Union:
925 72 Giulio Di Anastasio
926 85 Selvarani C
!https://redmine.auroville.org.in/attachments/download/9763/Union%20Menu.png!
927 1 Philippe May
928 1 Philippe May
929 85 Selvarani C
The new window for “Union” will appear:
930 1 Philippe May
931 86 Selvarani C
932 86 Selvarani C
!https://redmine.auroville.org.in/attachments/download/9764/Union%20Window.png!
933 73 Giulio Di Anastasio
934 74 Selvarani C
•	To make sure that the right geometry is generated by this process (“line” type, not “Multiline”, and similarly “Polygon” type, not “Multipolygon), we need to always keep the *Master shapefile* (e.g. Final-CZ-01-2021-02-05-LL84_V-BLDG-MHOL------E) *as Input layer*, and the Survey Area shapefile as Overlay Layer (e.g. WD-CZ-01-F-LL84_V-BLDG-MHOL------E).
935 73 Giulio Di Anastasio
•	(The output can be saved to a file, as the CRS should already be EPSG4326 – WGS84.)
936 73 Giulio Di Anastasio
937 74 Selvarani C
h2. 2.	Storing Shapefiles / Geopackages - (Selvarani, System 1)
938 74 Selvarani C
939 74 Selvarani C
Save the merged shape file in the correct location in Final folder as +D: > Survey > Zone-Survey Number > Final+ (eg . D: > Survey > GB-01 > Final)
940 73 Giulio Di Anastasio
Date in the name of Final Shapefile / Geopackage needs to be updated.
941 73 Giulio Di Anastasio
Once the merging operation is completed, the copy of Master shapefile / geopackage is deleted from the Temp folder.
942 73 Giulio Di Anastasio
943 74 Selvarani C
h2. 3.	Topology check of merged shapefiles
944 74 Selvarani C
945 73 Giulio Di Anastasio
The topology checker is applied again on the merged shapefiles / geopackages.
946 73 Giulio Di Anastasio
The “id_field” shall be removed from the attribute table.
947 73 Giulio Di Anastasio
948 74 Selvarani C
h2. 4.	Archive and replace the Master Shapefiles / Geopackages (Ram, System 4)
949 74 Selvarani C
950 73 Giulio Di Anastasio
Archive the previous master shapefiles / geopackages on system 4, and copy the new merged shapefiles / geopackages in its place.
951 74 Selvarani C
*Then delete the Merged Shapefile / Geopackage folder from System 1.
952 74 Selvarani C
*
953 73 Giulio Di Anastasio
954 74 Selvarani C
h2. 5.	Note about Shapefiles and Geopackages
955 73 Giulio Di Anastasio
956 73 Giulio Di Anastasio
All the above works are usually done using shapefile format, in QGIS latest version (3.16.3).
957 77 Selvarani C
The Geopackage export is done in QGis versions older than 3.12 (e.g. 3.4, 3.6, 3.8, 3.10) so that the lines are not saved as “Multilines” but as “Lines”and polygons are not saved as “Multipolygons” but as “Polygons”. This is very important to be notes, as Gisaf database does not accept the Multipolygon and Multiline geometry types.