Project

General

Profile

Wiki » History » Version 169

Selvarani C, 01/03/2021 16:47

1 99 Selvarani C
{{toc}}
2 99 Selvarani C
3 100 Selvarani C
h1. %{color:BLUE}  Wiki%
4 2 Philippe May
5 48 Philippe May
h2. About the Auroville CSR Geomatics Studio
6 48 Philippe May
7 48 Philippe May
We are a team working on geographical and related information with an engineering and scientific approach. The concept of *geomatics* is explained here: https://en.wikipedia.org/wiki/Geomatics.
8 48 Philippe May
9 48 Philippe May
We develop, maintain and publish data on this web site: https://gis.auroville.org.in.
10 48 Philippe May
11 48 Philippe May
h3. Team
12 48 Philippe May
13 48 Philippe May
Currently, the team consists of:
14 48 Philippe May
15 48 Philippe May
* Bala
16 48 Philippe May
17 48 Philippe May
* Giulio
18 48 Philippe May
19 48 Philippe May
* Philippe
20 48 Philippe May
21 48 Philippe May
* Raj
22 48 Philippe May
23 48 Philippe May
* Ram
24 48 Philippe May
25 48 Philippe May
* Selvarani
26 48 Philippe May
27 48 Philippe May
h3. Collaborations
28 1 Philippe May
29 91 Selvarani C
Quick report of the collaborations the Geomatics Team the areas of work.
30 45 Philippe May
31 89 Selvarani C
32 88 Selvarani C
h1. Collaborations
33 88 Selvarani C
34 88 Selvarani C
Collaborations/coordination with other other groups, etc
35 88 Selvarani C
 
36 88 Selvarani C
|_.CSR Geomatics Description of Activity / Research / Project|_.AA|_.AVF|_.LB|_.FAMC|_.WG|_.CSR|_.Talam|_.DST|
37 88 Selvarani C
|Topographic Survey of Auroville City Area in coordination with L’avenir d’Auroville: Matrimandir entire compound and Residential Zone Sector 1 completed, Sector 2 half completed|yes|||||||yes|
38 88 Selvarani C
|Topographic Survey of specific projects area: Vibrance, Wasteless, Cultural Zone along the Crown Road, Baraka, Gardens of Unexpected |  |  |  |  |  |  |  |  |
39 88 Selvarani C
| Collective wastewater treatment systems health check-up survey: 68 plants evaluated | yes |  |  |  | yes | yes |  | yes |
40 88 Selvarani C
| Manual weekly monitoring of water level in selected wells on Auroville land on approximately 50 wells (number fluctuates depending on local conditions) | yes |  |  |  | yes |  |  | yes |
41 88 Selvarani C
| Collection of rainfall data through manual raingauges distributed to Aurovilians: data received regularly from at least 7 raingauges |  |  |  |  | yes |  |  | yes |
42 88 Selvarani C
| Collection of weather data through automatic weather station installed at CSR: data collected every minute, stored in the database, and published online in real time |  |  |  |  | yes | yes |  | yes |
43 88 Selvarani C
| Collaboration with Land Board for survey of identified land boundary stones: collection of coordinates of Government boundary stones for georefering of cadastral maps |  | yes | yes |  |  |  |  |  |
44 88 Selvarani C
| Collaboration with AV Foundation for compilation of land ownership map: geographic land records as provided by AV Foundation, protected by login access |  | yes |  |  |  |  |  |  |
45 88 Selvarani C
| Collaboration with L’avenir d’Auroville for data sharing and coordinated system set-up: organisation of geographic data for unique online platform to enhance planning processes (initial stage) | yes |  |  |  |  |  |  |  |
46 88 Selvarani C
| Collaboration with L’avenir d’Auroville and FAMC for database links: structural work on procedures for linking databases already existing with various agencies | yes |  |  | yes |  |  |  |  |
47 88 Selvarani C
| Study of application of drone topographic survey in Auroville context: pilot test conducted on Residential Zone with outside drone survey agency | yes |  |  |  |  | yes |  | yes |
48 88 Selvarani C
| Study of open source software for drone topographic survey: testing of softwares like OpenDroneMap, MeshLab, CloudeCompare to evaluate work flow, processes and limitations |  |  |  |  |  | yes |  | yes |
49 88 Selvarani C
| Surface water modelling (ongoing): research on evaluation of surface water collection potential through automatic processing of field data |  |  |  |  |  | yes |  | yes |
50 88 Selvarani C
| Collaboration with TALAM on research on radio-transmission for automatic wells level monitoring (ongoing): evaluation of application of radio-transmitted signals for water-related monitoring and metering (selected borewells, flow meters) |  |  |  |  |  |  | yes | yes |
51 88 Selvarani C
| Organisation of data collected in Auroville in the past (ongoing): standardisation of data in geospatial database |  |  |  |  |  | yes |  |  |
52 88 Selvarani C
| Provision for basic statistical analyses of geographic-related data (indicators/dashboard) |  |  |  |  |  | yes |  | yes |
53 88 Selvarani C
| Publication of collected data in standardised format on web portal gis.auroville.org.in |  |  |  |  |  | yes |  | yes |
54 88 Selvarani C
55 88 Selvarani C
* AA : L'avenir d'Auroville
56 88 Selvarani C
* AVF : Auroville Foundation
57 88 Selvarani C
* LB : Auroville Land Board
58 88 Selvarani C
* FAMC : Auroville Funds and Assets Management Committee
59 88 Selvarani C
* WG : Auroville Water Group
60 88 Selvarani C
* CSR : Auroville Centre for Scientific Research
61 88 Selvarani C
* TALAM : a project under CSR
62 88 Selvarani C
* DST : Department of Science and Technology, Ministry of Science and Technology, New Delhi
63 88 Selvarani C
64 5 Philippe May
h2. Our workflow
65 2 Philippe May
66 92 Selvarani C
67 92 Selvarani C
h2. Surveys
68 92 Selvarani C
69 92 Selvarani C
Most field surveys are executed by our team of surveyors, using advanced DGPS equipment.
70 92 Selvarani C
Other surveyors might also be contracted.
71 92 Selvarani C
72 92 Selvarani C
73 92 Selvarani C
h2. CAD
74 92 Selvarani C
75 92 Selvarani C
The survey data are imported to a CAD software (Aurocad/Civil 3D).
76 93 Selvarani C
77 93 Selvarani C
h1. Editing shapefiles in Autocad
78 93 Selvarani C
79 93 Selvarani C
h2. FEATURES CREATION
80 93 Selvarani C
81 93 Selvarani C
1.	Assign a CRS to the drawing (TM-AUSPOS) (MAPCSLIBRARY command)
82 93 Selvarani C
2.	Create features in CAD (Points, lines, polygons)
83 93 Selvarani C
3.	Export shapefile (a) from CAD (Output > DWG to SDF) (Convert to LL84 – 3D)
84 93 Selvarani C
85 93 Selvarani C
h2. FEATURES IMPORT INTO DB
86 93 Selvarani C
87 93 Selvarani C
4.	Create zip file of the shapefile
88 93 Selvarani C
5.	Upload into the GISAF Shapefiles Basket
89 93 Selvarani C
6.	Import the shapefile into DB
90 93 Selvarani C
7.	Delete the shapefile from Local Machine
91 93 Selvarani C
92 93 Selvarani C
93 93 Selvarani C
h2. FEATURES EDITING
94 93 Selvarani C
95 93 Selvarani C
8.	Open the table in QGis
96 93 Selvarani C
9.	Save as a shapefile (b) in TM AUSPOS CRS
97 93 Selvarani C
10.	In CAD, open a new drawing and assign ASUPOS CRS
98 93 Selvarani C
11.	Import the shapefile (b) (MapImport) with all Object Data
99 93 Selvarani C
12.	Edit features
100 93 Selvarani C
13.	Export shapefile (a) from CAD (Output > DWG to SDF) with ONLY the id selected (Data Tab > Select Attributes > Object Data > Filename > id) (Convert to LL84 – 3D)
101 93 Selvarani C
102 93 Selvarani C
103 93 Selvarani C
h2. FEATURES IMPORT INTO DB
104 93 Selvarani C
105 93 Selvarani C
14.	Create zip file of the shapefile
106 93 Selvarani C
15.	Upload into the GISAF Shapefiles Basket
107 93 Selvarani C
16.	Import the shapefile into DB
108 93 Selvarani C
17.	Delete the shapefile from Local Machine
109 92 Selvarani C
110 96 Selvarani C
 
111 92 Selvarani C
112 92 Selvarani C
h2. QGis
113 92 Selvarani C
114 92 Selvarani C
h3. Conventions
115 2 Philippe May
116 3 Philippe May
117 94 Selvarani C
h1. Shapefiles
118 94 Selvarani C
119 94 Selvarani C
We work with "QGis":https://en.wikipedia.org/wiki/QGIS , a widely used open source, free software for working on maps and geographical data.
120 94 Selvarani C
121 94 Selvarani C
"Shapefile":https://en.wikipedia.org/wiki/Shapefile is a standard file format for mapping, that Gisaf can import. QGis can open and save *shapefiles*.
122 94 Selvarani C
123 94 Selvarani C
We have defined some simple norms about these shapefiles for integration with Gisaf.
124 94 Selvarani C
125 94 Selvarani C
126 94 Selvarani C
127 94 Selvarani C
h2. Coordinate system
128 94 Selvarani C
129 94 Selvarani C
We use CRS SRID 32644.
130 94 Selvarani C
131 94 Selvarani C
132 94 Selvarani C
133 94 Selvarani C
h2. Column (attributes)
134 94 Selvarani C
135 94 Selvarani C
* All objects in a shapefile (layer) have a unique identifier named "id": numerical value.
136 94 Selvarani C
137 94 Selvarani C
138 94 Selvarani C
139 94 Selvarani C
h3. Field (attribute) names
140 94 Selvarani C
141 94 Selvarani C
* All fields are lower case (-UpperCase-, use: @lowercase@)
142 94 Selvarani C
143 94 Selvarani C
* They do not contain spaces, but underscores (-not this-, use: @but_that@)
144 94 Selvarani C
145 94 Selvarani C
* The field names cannot have more than 8 characters (-this_is_too_long-, use: @this_short@)
146 94 Selvarani C
147 94 Selvarani C
148 94 Selvarani C
h3. Attribute types
149 94 Selvarani C
150 94 Selvarani C
* The dates are written in ISO format: @YYYY-MM-DD@ (eg. @1968-02-25@)
151 94 Selvarani C
152 94 Selvarani C
h3. Fields to remove
153 94 Selvarani C
154 94 Selvarani C
* Eventually, remove the fields containing the coordinates: northing, easting, elevation, latitude, longitude, area, length, etc (these might be present when the data is imported from speadsheet)
155 94 Selvarani C
156 94 Selvarani C
h2. Foreign keys
157 94 Selvarani C
158 94 Selvarani C
We often deal with labels or categories of objects in a layer.
159 94 Selvarani C
160 94 Selvarani C
A common use case to explain: Alice creates a layer of stones, and wants to *tag* each stone with a rating: these are picked from a list of choices, like: _Beautiful_, _Interesting_, _Pretty_, etc.
161 94 Selvarani C
162 94 Selvarani C
For these kind of attribute: define a column like @rating_id@ (_something_ ending with @_id@) as a numerical value, and assign values 1, 2, 3, etc. The text is defined in another table (typically a CSV file), that looks like:
163 94 Selvarani C
164 94 Selvarani C
|_. id |_. name |
165 94 Selvarani C
| 1 | Beautiful |
166 94 Selvarani C
| 2 | Interesting |
167 94 Selvarani C
| 3 | Pretty |
168 94 Selvarani C
169 94 Selvarani C
h2. Code
170 94 Selvarani C
171 94 Selvarani C
We have defined a standard set of codes, that defines the type of data. They can be found here (TODO: add link).
172 94 Selvarani C
173 94 Selvarani C
Add a column @code_name@, matching with one the code, eg. @V25@ for TDEF.
174 94 Selvarani C
175 94 Selvarani C
h2. Surveyor
176 94 Selvarani C
177 94 Selvarani C
We keep a record of the people who realized the surveys (the _surveyors_).
178 94 Selvarani C
The shapefiles must contain an attribute @srvyr_id@, which refers to this table (TODO: add link).
179 94 Selvarani C
180 94 Selvarani C
h2. Accuracy
181 94 Selvarani C
182 94 Selvarani C
We keep a record of the accuracy of the surveys.
183 94 Selvarani C
The shapefiles must contain an attribute @accur_id@, which refers to this table (TODO: add link).
184 94 Selvarani C
185 94 Selvarani C
186 94 Selvarani C
h2. Date of survey
187 94 Selvarani C
188 94 Selvarani C
As nothing is absolutely permanent, it's also important to keep track of the date of the surveys: the shapefiles must contain an attribute @date@.
189 94 Selvarani C
190 94 Selvarani C
191 3 Philippe May
h2. Working with Gisaf
192 3 Philippe May
193 6 Philippe May
h3. Survey data
194 6 Philippe May
195 12 Philippe May
Raw survey data are contained in CSV files, typically downloaded from surveying instruments.
196 12 Philippe May
197 144 Selvarani C
See more information of the process for the survey data (including a flow diagram)
198 12 Philippe May
199 97 Selvarani C
h1. Survey data
200 97 Selvarani C
201 97 Selvarani C
h2. Workflow summary
202 97 Selvarani C
203 147 Selvarani C
p=. !https://redmine.auroville.org.in/attachments/download/9788/Data_workflow2.odg!
204 97 Selvarani C
205 97 Selvarani C
h2. Import basket
206 97 Selvarani C
207 97 Selvarani C
Gisaf provides an "import basket" dedicated for raw survey data, which is generated by land survey equipment (Leica's Total Station and RTK). 
208 97 Selvarani C
209 97 Selvarani C
These are CSV files, like:
210 97 Selvarani C
<pre>
211 97 Selvarani C
100081,370633.969,1327742.157,51.187,,,
212 97 Selvarani C
100083,370628.876,1327702.913,51.565,T52,,
213 97 Selvarani C
100082,370628.729,1327720.019,51.261,T52,,
214 97 Selvarani C
100081,370633.969,1327742.154,51.179,,,
215 97 Selvarani C
100083,370628.876,1327702.913,51.565,T52,,
216 97 Selvarani C
20800,370633.969,1327742.154,51.180,,,
217 97 Selvarani C
20801,370618.795,1327713.172,52.817,E30,,
218 97 Selvarani C
20802,370623.674,1327711.436,51.283,B35,,
219 97 Selvarani C
20803,370619.314,1327713.407,51.383,B35,,
220 97 Selvarani C
</pre>
221 97 Selvarani C
222 97 Selvarani C
Each category (5th column) must be defined in the Category table (see [[Categories]]).
223 97 Selvarani C
224 97 Selvarani C
225 97 Selvarani C
h2. Organization of the raw survey data basket
226 97 Selvarani C
227 97 Selvarani C
The basket should be organized in a directory structure:
228 97 Selvarani C
229 97 Selvarani C
- Project name (these can be themselves put in a hierarchy of (sub)directories)
230 97 Selvarani C
231 97 Selvarani C
- Surveyor's organization
232 97 Selvarani C
233 97 Selvarani C
- Equipment (eg. TS, RTK)
234 97 Selvarani C
235 97 Selvarani C
- Survey files (eg. @Our_project-Some_comment-2018-02-23.txt@)
236 97 Selvarani C
237 97 Selvarani C
h3. Format of the survey file names
238 97 Selvarani C
239 97 Selvarani C
<pre>
240 97 Selvarani C
Our_project-Some_comment-2018-02-23.txt
241 97 Selvarani C
</pre>
242 97 Selvarani C
243 97 Selvarani C
The date of the survey follows the ISO date standard: @YYYY-MM-DD@.
244 97 Selvarani C
245 97 Selvarani C
246 97 Selvarani C
h2. Import to the database
247 97 Selvarani C
248 97 Selvarani C
When importing raw survey data files to the database, Gisaf does 2 steps as described below.
249 97 Selvarani C
It's worth noting that, in this process, no reprojection is done.
250 97 Selvarani C
251 97 Selvarani C
252 97 Selvarani C
h3. Feed the raw_survey table
253 97 Selvarani C
254 97 Selvarani C
Each point of the imported raw survey data file is inserted to the raw_survey table:
255 97 Selvarani C
256 97 Selvarani C
# Creation of a Point geometry: the raw_survey table has a geometry column for a single point (@geom@) with x,y and z coordinates
257 97 Selvarani C
# Save the @id@ of the original point *to the @orig_id@ column*
258 97 Selvarani C
# *A unique @id@ is computed* from the following fields: @id@, @project@, @equipment@, @date@
259 97 Selvarani C
# The project is saved in the @project_id@ column
260 97 Selvarani C
# The surveyor identification in @srvyr_id@
261 97 Selvarani C
# The date of survey is saved in the @date@ column
262 97 Selvarani C
# The accuracy is tagged in the @accur_id@, according to a mapping defined in the @accuracyequimentsurveyormapping@ table, which depends on the surveyor and equipment
263 97 Selvarani C
# The category of the point
264 97 Selvarani C
265 97 Selvarani C
266 97 Selvarani C
h3. Feed the @RAW_V_*@ tables
267 97 Selvarani C
268 97 Selvarani C
From the @raw_survey@ table, each point is then copied to its respective @RAW_V_@ table, with basically the same information.
269 97 Selvarani C
270 97 Selvarani C
These tables (which should be created manually or with the admin notebook called @create_tables@, as of today's writing), only contain points.
271 97 Selvarani C
272 97 Selvarani C
The project is saved along: see below.
273 97 Selvarani C
274 97 Selvarani C
275 97 Selvarani C
h2. Import the points
276 97 Selvarani C
277 97 Selvarani C
For categories that define points (opposite to lines and polygons, which require _line work_ carried over in CAD or in a GIS software, see [[Line work]]), the points can be imported automatically to their final destination: the @V_*@ tables.
278 97 Selvarani C
279 97 Selvarani C
Note: in this process, the geometries are reprojected.
280 97 Selvarani C
281 97 Selvarani C
282 97 Selvarani C
h3. Auto import of the points
283 97 Selvarani C
284 97 Selvarani C
The points found in the @RAW_V_*@ tables can be imported automatically, project per project, the project page of the admin interface.
285 97 Selvarani C
286 97 Selvarani C
287 97 Selvarani C
h3. Import of the line work (lines and polygons)
288 97 Selvarani C
289 142 Selvarani C
h2. See [[Line work]] - *%{color:RED} to be written%*
290 97 Selvarani C
291 97 Selvarani C
The shapefiles generated manually (line work) should be put in the project's basket, and imported from it.
292 97 Selvarani C
293 148 Selvarani C
h1. Categories
294 12 Philippe May
295 102 Selvarani C
The categories define the types of the geographical features and they are mapped according to ISO standard layer naming conventions: see https://www.nationalcadstandard.org/ncs5/pdfs/ncs5_clg_lnf.pdf
296 101 Selvarani C
297 101 Selvarani C
Gisaf uses:
298 101 Selvarani C
299 101 Selvarani C
* a table @category@ where the layers are defined
300 101 Selvarani C
* a table per category
301 101 Selvarani C
302 101 Selvarani C
h2. Fields for the categories
303 101 Selvarani C
304 101 Selvarani C
TODO
305 101 Selvarani C
306 101 Selvarani C
h2. Creation of the RAW_* tables
307 101 Selvarani C
308 101 Selvarani C
This step must be done manually (as of today's date of writing).
309 101 Selvarani C
310 6 Philippe May
311 1 Philippe May
h3. QGis: work on shapefiles
312 1 Philippe May
313 142 Selvarani C
h2. Go to [[shapefiles]] - *%{color:RED} to be written%*
314 5 Philippe May
315 103 Selvarani C
h1. Data analysis
316 18 Philippe May
317 103 Selvarani C
We use "Jupyter":https://jupyter.org , "Pandas":https://pandas.pydata.org/ and "GeoPandas":http://geopandas.org/ , accessible at http://gis.auroville.org.in/notebooks .
318 18 Philippe May
319 103 Selvarani C
For integration in the processes (execution of notebooks), there's "papermill":https://github.com/nteract/papermill . Systemd "timers":https://wiki.archlinux.org/index.php/Systemd/Timers are used to automatically schedule the notebooks on the server, ie. for the dashboards.
320 1 Philippe May
321 103 Selvarani C
There's a dedicated virtual machine for Jupyter, accessible from our local network at @jupyter.csr.av@.
322 103 Selvarani C
323 103 Selvarani C
h2. Organization of notebooks
324 103 Selvarani C
325 103 Selvarani C
The setup is organized in 2 parts, that are run with 2 instances of Jupyter for security reasons.
326 103 Selvarani C
327 103 Selvarani C
h3. Admin
328 103 Selvarani C
329 103 Selvarani C
The notebooks in the admin are mostly for maintenance: operations on the database, etc.
330 103 Selvarani C
331 103 Selvarani C
h3. Users
332 103 Selvarani C
333 103 Selvarani C
The notebooks are organized in folders, all under Gisaf's source code git repository, except the "Sandbox" one.
334 103 Selvarani C
335 103 Selvarani C
This notebook server connects to the database with a specific user (@jupyter@), which has been set on the database server with permissions to read all data (@readonly@) plus has write access to some tables dedicated to store analysis results.
336 103 Selvarani C
337 103 Selvarani C
h2. Integration with Gisaf
338 103 Selvarani C
339 103 Selvarani C
The notebook in @Templates@ demonstrates the usage of notebook in relation with Gisaf: mostly, how to use the @gisad.ipynb_tools@ module to access Gisaf models and the data from the database.
340 103 Selvarani C
341 103 Selvarani C
This module is part of gisaf: https://redmine.auroville.org.in/projects/gisaf/repository/revisions/master/entry/gisaf/ipynb_tools.py
342 103 Selvarani C
343 103 Selvarani C
h2. References
344 103 Selvarani C
345 103 Selvarani C
h3. Geopandas
346 103 Selvarani C
347 103 Selvarani C
Some nice examples of processing, using water shed and rain: https://geohackweek.github.io/vector/06-geopandas-advanced/
348 103 Selvarani C
349 103 Selvarani C
h3. Integration
350 103 Selvarani C
351 103 Selvarani C
A good example of how a company has integrated the same tools: https://medium.com/netflix-techblog/scheduling-notebooks-348e6c14cfd6
352 103 Selvarani C
353 103 Selvarani C
h2. Other docs
354 103 Selvarani C
355 103 Selvarani C
h3. From Swathi
356 5 Philippe May
357 5 Philippe May
h2. Hosting
358 5 Philippe May
359 5 Philippe May
The team is located in the CSR of Auroville.
360 5 Philippe May
361 104 Selvarani C
We have setup a server for hosting the software and database. CSR_server for technical information about the setup.
362 1 Philippe May
363 104 Selvarani C
h1. CSR server
364 1 Philippe May
365 104 Selvarani C
dream.csr.av (192.168.0.12)
366 104 Selvarani C
367 104 Selvarani C
- Debian 9
368 104 Selvarani C
- Xen hypervisor
369 104 Selvarani C
- libvirt for the orchestration of VMs
370 104 Selvarani C
- management with ansible
371 104 Selvarani C
372 104 Selvarani C
373 104 Selvarani C
h2. Dom0
374 104 Selvarani C
375 104 Selvarani C
h3. Installation
376 104 Selvarani C
377 104 Selvarani C
Found minor issues with the installation (eg. issues with HDDs, Dell EFI boot).
378 104 Selvarani C
This document starts from a working Debian Xen server.
379 104 Selvarani C
380 104 Selvarani C
-Installed on a mirror of 2*2TB drives with btrfs.-
381 104 Selvarani C
382 104 Selvarani C
Update (see #7156): re-installed the OS on /dev/sdc2, ext4, without RAID/LVM.
383 104 Selvarani C
384 104 Selvarani C
h3. Storage for domUs
385 104 Selvarani C
386 104 Selvarani C
A LVM dream.csr has been created for the domUs.
387 104 Selvarani C
388 104 Selvarani C
389 104 Selvarani C
h3. Networking
390 104 Selvarani C
391 104 Selvarani C
With systemd.networkd: bridge "br0" to the physical ethernet.
392 104 Selvarani C
393 104 Selvarani C
h3. Create a VM
394 104 Selvarani C
395 104 Selvarani C
Adjust the parameters from:
396 104 Selvarani C
397 104 Selvarani C
<pre>
398 104 Selvarani C
export vm_name=infra.csr.av
399 104 Selvarani C
export vm_ip_addr=172.16.0.3
400 104 Selvarani C
export vm_password=foobar
401 104 Selvarani C
</pre>
402 104 Selvarani C
403 104 Selvarani C
Create the domU:
404 104 Selvarani C
405 104 Selvarani C
<pre>
406 104 Selvarani C
root@dream:~# xen-create-image --broadcast=172.16.0.255 --dist=stretch --fs=ext4 --gateway=172.16.0.1 --hostname ${vm_name} --ip=${vm_ip_addr} --lvm=dream.csr --maxmem=512M --memory=256M --mirror=http://ftp.de.debian.org/debian/ --netmask=255.255.255.0 --password=${vm_password} --size=10G --swap=1G --vcpus=1 --bridge=br0
407 104 Selvarani C
</pre>
408 104 Selvarani C
409 104 Selvarani C
Note that IP address will be set in the VM, and the vm_ip_addr isn't actually used.
410 104 Selvarani C
411 104 Selvarani C
h2. DomUs
412 104 Selvarani C
413 104 Selvarani C
h3. Migrate XL to libvirt
414 104 Selvarani C
415 104 Selvarani C
After creation using xen-create-image, migrate the definition of the domU to libvirt:
416 104 Selvarani C
417 104 Selvarani C
<pre>
418 104 Selvarani C
virsh -c xen:/// domxml-from-native xen-xm /etc/xen/${vm_name}.cfg > /tmp/${vm_name}.xml
419 104 Selvarani C
virsh define /tmp/${vm_name}.xml
420 104 Selvarani C
</pre>
421 104 Selvarani C
422 104 Selvarani C
423 104 Selvarani C
From this point onward, one can log out from the dom0's console and use virsh or "Virtual Machine Manager" from your computer to administer the VM, eg:
424 104 Selvarani C
425 104 Selvarani C
* Set "Auto start" on dom0 boot
426 104 Selvarani C
* Set memory limits, etc
427 104 Selvarani C
428 104 Selvarani C
h3. Start the domU
429 104 Selvarani C
430 104 Selvarani C
Use "Virtual Machine Manager" or the command:
431 104 Selvarani C
432 104 Selvarani C
<pre>
433 104 Selvarani C
virsh -c xen+ssh://root@dream.csr.av/ start ${vm_name}
434 104 Selvarani C
</pre>
435 104 Selvarani C
436 104 Selvarani C
437 104 Selvarani C
h3. To do in a libvirt shell
438 104 Selvarani C
439 104 Selvarani C
Start a libvirt shell, with "Virtual Machine Manager" or with the command:
440 104 Selvarani C
441 104 Selvarani C
<pre>
442 104 Selvarani C
virsh -c xen+ssh://root@dream.csr.av/ console ${vm_name}
443 104 Selvarani C
</pre>
444 104 Selvarani C
445 104 Selvarani C
Log in as root in the libvirt console.
446 104 Selvarani C
447 104 Selvarani C
448 104 Selvarani C
h4. Network config
449 104 Selvarani C
450 104 Selvarani C
Add @/etc/systemd/network/50-wired.network@ (adjust the IP):
451 104 Selvarani C
452 104 Selvarani C
<pre>
453 104 Selvarani C
[Match]
454 104 Selvarani C
Name=eth0
455 104 Selvarani C
456 104 Selvarani C
[Network]
457 104 Selvarani C
Address=192.168.0.14/24
458 104 Selvarani C
Gateway=192.168.0.10
459 104 Selvarani C
DNS=192.168.0.10
460 104 Selvarani C
</pre>
461 104 Selvarani C
462 104 Selvarani C
Then:
463 104 Selvarani C
<pre>
464 104 Selvarani C
systemctl enable --now systemd-networkd.socket
465 104 Selvarani C
systemctl enable --now systemd-networkd.service
466 104 Selvarani C
systemctl disable --now networking.service
467 104 Selvarani C
</pre>
468 104 Selvarani C
469 104 Selvarani C
h4. Post-install ssh config
470 104 Selvarani C
471 104 Selvarani C
Allow ssh root login with password in @/etc/ssh/sshd_config@:
472 104 Selvarani C
473 104 Selvarani C
<pre>
474 104 Selvarani C
sed -i -e 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
475 104 Selvarani C
systemctl restart ssh.service
476 104 Selvarani C
</pre>
477 104 Selvarani C
478 104 Selvarani C
From this point onwards, one can close the console session opened via @libvirt@.
479 104 Selvarani C
480 104 Selvarani C
h3. DNS
481 104 Selvarani C
482 104 Selvarani C
Log in to the local DNS server with:
483 104 Selvarani C
484 104 Selvarani C
<pre>
485 104 Selvarani C
ssh root@infra.csr.av
486 104 Selvarani C
</pre>
487 104 Selvarani C
488 104 Selvarani C
Update the 2 files in @/etc/bind/zones@ (@zones/db.csr.av@ @zones/db.192.168.0@) and reload the DNS with:
489 104 Selvarani C
<pre>
490 104 Selvarani C
rndc reload
491 104 Selvarani C
</pre>
492 104 Selvarani C
493 104 Selvarani C
h3. DNS
494 104 Selvarani C
495 104 Selvarani C
Log in to the local DNS server, *update the 2 files in @/etc/bind9/zones@*, and run:
496 104 Selvarani C
<pre>
497 104 Selvarani C
rndc reload
498 104 Selvarani C
</pre>
499 104 Selvarani C
500 104 Selvarani C
h3. Allow yourself to log in easily with your ssh key
501 104 Selvarani C
502 104 Selvarani C
Copy your ssh key to the domU: run from your own computer:
503 104 Selvarani C
<pre>
504 104 Selvarani C
ssh-copy-id root@${ip_addr}
505 104 Selvarani C
</pre>
506 104 Selvarani C
507 104 Selvarani C
508 104 Selvarani C
h2. Ansible
509 104 Selvarani C
510 104 Selvarani C
Using the Ansible project developed in Blue Light: https://redmine.bluelightav.org/projects/ansible
511 104 Selvarani C
512 104 Selvarani C
h3. Prepare the host
513 104 Selvarani C
514 104 Selvarani C
Install Python
515 104 Selvarani C
516 104 Selvarani C
<pre>
517 104 Selvarani C
apt install -y python
518 104 Selvarani C
</pre>
519 104 Selvarani C
520 104 Selvarani C
521 104 Selvarani C
h2. Database
522 104 Selvarani C
523 104 Selvarani C
The Postgis database runs on its specific domU (gisdb.csr.av, 192.168.0.18).
524 104 Selvarani C
525 104 Selvarani C
h3. Installation
526 104 Selvarani C
527 104 Selvarani C
After installing the Postgis package (eg. assign the "postgis" Ansible's role), follow up to [[Db-support]]
528 104 Selvarani C
529 104 Selvarani C
530 104 Selvarani C
h2. Jupyter
531 104 Selvarani C
532 104 Selvarani C
The Jupyter server runs on its specific domU (jupyter.csr.av, 192.168.0.19).
533 104 Selvarani C
534 104 Selvarani C
h3. Installation
535 104 Selvarani C
536 104 Selvarani C
See #6990 .
537 105 Selvarani C
538 150 Selvarani C
h1. Backup and restoration of the database
539 1 Philippe May
540 105 Selvarani C
h1. Database
541 105 Selvarani C
542 105 Selvarani C
h2. Troubleshooting
543 105 Selvarani C
544 105 Selvarani C
h3. Layers missing in the map's tree
545 105 Selvarani C
546 105 Selvarani C
Gisaf relies on counting features through Postgres statistics collector subsystem.
547 105 Selvarani C
548 105 Selvarani C
In case the server is restarted *dirty* (eg. without clean shutdown), then the count of the tables might be wrong or just 0, leaving the layers apparently empty and thus not even appearing.
549 105 Selvarani C
550 105 Selvarani C
The fix is as easy as:
551 105 Selvarani C
552 105 Selvarani C
<pre>
553 105 Selvarani C
sudo -u postgres psql avgis -c VACUUM
554 105 Selvarani C
</pre>
555 105 Selvarani C
556 105 Selvarani C
h2. Installation
557 105 Selvarani C
558 105 Selvarani C
This documentation assumes that the Postgis package has been installed (see [[CSR_server#Database]]).
559 105 Selvarani C
560 105 Selvarani C
h3. Configure the server
561 105 Selvarani C
562 105 Selvarani C
h4. Allow connections from other hosts in the local network
563 105 Selvarani C
564 105 Selvarani C
Set the server to listen to addresses, set listen_addresses to @*@ in @/etc/postgresql/9.6/main/postgresql.conf@.
565 105 Selvarani C
566 105 Selvarani C
567 105 Selvarani C
Allow the connections, add in @/etc/postgresql/9.6/main/pg_hba.conf@:
568 105 Selvarani C
569 105 Selvarani C
<pre>
570 105 Selvarani C
host all all 192.168.0.0/24 md5
571 105 Selvarani C
</pre>
572 105 Selvarani C
573 105 Selvarani C
574 105 Selvarani C
h2. Creation of the database
575 105 Selvarani C
576 105 Selvarani C
As @postgres@ user:
577 105 Selvarani C
578 105 Selvarani C
<pre>
579 105 Selvarani C
createdb -E utf8 -T template0 avgis
580 105 Selvarani C
</pre>
581 105 Selvarani C
582 105 Selvarani C
h2. Backups
583 105 Selvarani C
584 105 Selvarani C
h3. Primary
585 105 Selvarani C
586 105 Selvarani C
The database is backed up every day at midnight. The dump file is located in @/var/backups/postgres/@.
587 105 Selvarani C
588 105 Selvarani C
h3. Secondary
589 105 Selvarani C
590 105 Selvarani C
There are other backups (daily, weekly, monthly) thanks to Debian package @autopostgresqlbackup@), located (default) in @/var/lib/autopostgresqlbackup@.
591 105 Selvarani C
592 105 Selvarani C
h3. Tertiary (dom0)
593 105 Selvarani C
594 105 Selvarani C
The whole virtual machine is backed up by BackupNinja on the "dom0" controller, using:
595 105 Selvarani C
- rdiff backups every day
596 105 Selvarani C
- tar files on Saturdays.
597 105 Selvarani C
598 105 Selvarani C
See @/etc/backups.d@ on the dom0 (192.168.0.12).
599 105 Selvarani C
600 105 Selvarani C
h3. Remote
601 105 Selvarani C
602 105 Selvarani C
TODO: remote backup.
603 105 Selvarani C
604 105 Selvarani C
605 105 Selvarani C
h2. Restoration
606 105 Selvarani C
607 105 Selvarani C
If the VM is not shutdown properly, there's a chance that the database is corrupt, and needs to be restored from one of the backups.
608 105 Selvarani C
609 105 Selvarani C
After the restoration, restart gisaf:
610 105 Selvarani C
<pre>
611 105 Selvarani C
systemctl restart uwsgi.service
612 105 Selvarani C
</pre>
613 105 Selvarani C
614 105 Selvarani C
h3. From primary backup
615 105 Selvarani C
616 105 Selvarani C
*Note*: the roles aren't restored with this method.
617 105 Selvarani C
618 105 Selvarani C
With user @postgres@:
619 105 Selvarani C
<pre>
620 105 Selvarani C
# Optionally, rename the corrupt database (selecting a name for a database like "avgis_c2")...
621 105 Selvarani C
psql -c "ALTER DATABASE avgis RENAME TO avgis_c2;"
622 105 Selvarani C
# ... or drop the existing database
623 105 Selvarani C
psql -c "drop database avgis;"
624 105 Selvarani C
# Create a new database:
625 105 Selvarani C
createdb -E utf8 -T template0 avgis
626 105 Selvarani C
# Restore the database
627 105 Selvarani C
pg_restore -d avgis /var/backups/postgres/avgis.pg_dump
628 105 Selvarani C
</pre>
629 105 Selvarani C
630 105 Selvarani C
h3. From secondary backup
631 105 Selvarani C
632 105 Selvarani C
@autopostgresqlbackup@ backs up the roles in @postgres_globals@.
633 105 Selvarani C
634 105 Selvarani C
<pre>
635 105 Selvarani C
zcat /var/lib/autopostgresqlbackup/daily/postgres_globals/postgres_globals_2018-10-24_06h25m.Wednesday.sql.gz | psql
636 105 Selvarani C
zcat /var/lib/autopostgresqlbackup/daily/avgis/avgis_2018-10-24_06h25m.Wednesday.sql.gz | psql
637 105 Selvarani C
</pre>
638 17 Philippe May
639 17 Philippe May
h2. Gear
640 17 Philippe May
641 17 Philippe May
h3. Survey equipment
642 17 Philippe May
643 143 Selvarani C
h2. See [[survey equipment]] - *%{color:RED} to be written%*
644 17 Philippe May
645 152 Selvarani C
h1. Weather station
646 17 Philippe May
647 152 Selvarani C
648 11 Philippe May
649 106 Selvarani C
h1. Ambient Weather weather station
650 106 Selvarani C
651 106 Selvarani C
We have purchased a WS2902A weather station (https://www.ambientweather.com/amws2902.html).
652 106 Selvarani C
Firmware version: 4.0.2.
653 106 Selvarani C
654 106 Selvarani C
h2. Manual
655 106 Selvarani C
656 106 Selvarani C
The operating manual of the weather station can be found at https://p10.secure.hostingprod.com/@site.ambientweatherstore.com/ssl/Manuals/WS-2902C.pdf
657 106 Selvarani C
658 106 Selvarani C
h2. Connection
659 106 Selvarani C
660 106 Selvarani C
h3. Wifi
661 106 Selvarani C
662 106 Selvarani C
Set up the wifi of the console using the "Ambient Tool" phone application. IP address given by DHCP on the router is: 192.168.1.101
663 106 Selvarani C
664 106 Selvarani C
h3. Local communication
665 106 Selvarani C
666 106 Selvarani C
Fail so far: the only exposed port is TCP/45000. Telnet doesn't show any activity. Nothing found on Internet on this protocol.
667 106 Selvarani C
668 106 Selvarani C
One interesting project may come, hijacking the connection to cloud services: https://www.wxforum.net/index.php?topic=35033.0
669 106 Selvarani C
670 106 Selvarani C
671 106 Selvarani C
h3. Cloud connection
672 106 Selvarani C
673 106 Selvarani C
We'll create an account on AmbientWeather.net (and eventually on WUnderground.net and/or weathercloud.net), and:
674 106 Selvarani C
675 106 Selvarani C
* have the console upload data to there
676 106 Selvarani C
* gisaf to retrieve our WS data from there
677 106 Selvarani C
678 106 Selvarani C
h1. Notes on Console
679 106 Selvarani C
680 106 Selvarani C
The daily rainfall data displayed in  the console resets at 00.30 every night
681 11 Philippe May
682 7 Philippe May
h2. Plan for future
683 7 Philippe May
684 154 Selvarani C
Beside living well, 
685 8 Philippe May
686 107 Selvarani C
h1. Plan
687 107 Selvarani C
688 107 Selvarani C
Some interesting projects that might be integrated:
689 107 Selvarani C
690 107 Selvarani C
* https://github.com/Oslandia/albion : Build 3D geological model from wells information
691 107 Selvarani C
692 49 Philippe May
h2. Other
693 1 Philippe May
694 155 Selvarani C
695 49 Philippe May
696 108 Selvarani C
h1. GDAL (OGR) tools
697 108 Selvarani C
698 108 Selvarani C
"GDAL":https://gdal.org/ is a translator library for raster and vector geospatial data formats. It is used by many software (including QGIS and many other open source ones, including Gisaf). Some command line utilities are supplied, like:
699 108 Selvarani C
700 108 Selvarani C
* @ogr2ogr@ can easily convert one data format to another
701 108 Selvarani C
* @ogrinfo@ displays information about files.
702 108 Selvarani C
703 108 Selvarani C
h2. Using Windows
704 108 Selvarani C
705 108 Selvarani C
On a computer with Windows and GQIS installed:
706 108 Selvarani C
707 108 Selvarani C
1. Open a command line console (eg. <Windows Key> to display the Start menu, then just type @cmd@ and <Enter>)
708 108 Selvarani C
2. In the console window, type (adjust with the QGIS version and location, this seems to be the standard one):
709 108 Selvarani C
710 108 Selvarani C
<pre>
711 108 Selvarani C
"c:\Program Files\QGis 3.10\OSGeo4W.bat"
712 108 Selvarani C
</pre>
713 108 Selvarani C
714 108 Selvarani C
3. GDAL utilities can be used: @ogr2ogr@, etc.
715 108 Selvarani C
716 108 Selvarani C
h2. Example: convert Geopackage to Shapefiles
717 108 Selvarani C
718 108 Selvarani C
Output the content of the geopackage @9wdoogfr_2019-11-13_12_26_07.gpkg@ to the folder @shapefiles@:
719 108 Selvarani C
720 108 Selvarani C
<pre>
721 108 Selvarani C
ogr2ogr -progress -f "ESRI Shapefile" shapefiles 9wdoogfr_2019-11-13_12_26_07.gpkg
722 108 Selvarani C
</pre>
723 108 Selvarani C
724 108 Selvarani C
To output the content of the geopackage @9wdoogfr_2019-11-13_12_26_07.gpkg@ to the @root@ folder:
725 108 Selvarani C
726 108 Selvarani C
<pre>
727 108 Selvarani C
ogr2ogr -progress -f "ESRI Shapefile" c:\shapefiles 9wdoogfr_2019-11-13_12_26_07.gpkg
728 108 Selvarani C
</pre>
729 108 Selvarani C
730 108 Selvarani C
h2. With reprojection
731 108 Selvarani C
732 108 Selvarani C
Same as above, reprojecting to UTM44N: 
733 108 Selvarani C
734 108 Selvarani C
<pre>
735 108 Selvarani C
ogr2ogr -progress -f "ESRI Shapefile" -t_srs EPSG:32644 c:\shapefiles 9wdoogfr_2019-11-13_12_26_07.gpkg
736 108 Selvarani C
</pre>
737 108 Selvarani C
738 8 Philippe May
h2. Links and references
739 8 Philippe May
740 156 Selvarani C
741 16 Philippe May
742 109 Selvarani C
h1. Links
743 109 Selvarani C
744 109 Selvarani C
h2. Water management
745 109 Selvarani C
746 109 Selvarani C
h3. Modflow
747 109 Selvarani C
748 109 Selvarani C
* https://water.usgs.gov/ogw/modflow/
749 109 Selvarani C
750 109 Selvarani C
The reference software for underground water modelling and simulation.
751 109 Selvarani C
In conjunction with flopy (https://water.usgs.gov/ogw/flopy/) and Jupyter (https://jupyter.org/), it provides a _relatively easy_ to use interface.
752 109 Selvarani C
753 109 Selvarani C
754 109 Selvarani C
h3. Freewat
755 109 Selvarani C
756 109 Selvarani C
* http://www.freewat.eu/project
757 109 Selvarani C
758 109 Selvarani C
This project is partly based on modflow, and integrates with QGis.
759 109 Selvarani C
760 109 Selvarani C
h2. QGIS
761 109 Selvarani C
762 109 Selvarani C
* https://www.qgis.org/en/site/
763 109 Selvarani C
764 109 Selvarani C
* Tools for Geology
765 109 Selvarani C
Construction of geological cross sections in QGIS - http://www.geokincern.com/?p=1452
766 109 Selvarani C
767 109 Selvarani C
768 109 Selvarani C
h2. Autocad
769 109 Selvarani C
770 109 Selvarani C
* Overview of Converting Geospatial Data to Drawing Objects: 
771 109 Selvarani C
http://docs.autodesk.com/CIV3D/2013/ENU/index.html?url=filesMAPC3D/GUID-C38FD485-3CC2-4B52-8264-0D8C0F45422B.htm,topicNumber=MAPC3Dd30e41809
772 109 Selvarani C
773 109 Selvarani C
* CAD-DB connection:
774 109 Selvarani C
https://knowledge.autodesk.com/support/autocad-civil-3d/learn-explore/caas/video/youtube/watch-v-AQoB--nyUJA.html
775 109 Selvarani C
776 109 Selvarani C
777 109 Selvarani C
h2. Orfeo
778 109 Selvarani C
779 109 Selvarani C
* https://www.orfeo-toolbox.org/
780 109 Selvarani C
781 109 Selvarani C
Remote sensing
782 16 Philippe May
783 16 Philippe May
h2. Old docs
784 16 Philippe May
785 157 Selvarani C
[[Shapefiles]] - *%{color:RED} to be written%*
786 158 Selvarani C
787 16 Philippe May
788 110 Selvarani C
h1. Data (measurements auxiliary tables)
789 110 Selvarani C
790 110 Selvarani C
Besides the importation of [[shapefiles]], Gisaf can import non-geophical information: auxiliary data (typically categories like the list of locations names, well types, etc), and temporal informations (well levels, etc).
791 110 Selvarani C
792 110 Selvarani C
h2. Command line
793 110 Selvarani C
794 110 Selvarani C
The @import_to_db.py@ script imports files, fetched from a set of URLs (typically, in the Redmine Files section of this project), formats and pre-process, and imports to the database.
795 110 Selvarani C
796 110 Selvarani C
@import_to_db.py@ is a support tool, that is planned to be integrated with the web interface.
797 110 Selvarani C
798 110 Selvarani C
Import all with:
799 110 Selvarani C
800 110 Selvarani C
<pre>
801 110 Selvarani C
phil@phil-mbp:~/BlueLight/gisaf_src/gisaf$ python import_to_db.py
802 110 Selvarani C
</pre>
803 110 Selvarani C
804 110 Selvarani C
The script currently accepts an argument for filtering the URLs to import.
805 110 Selvarani C
806 13 Philippe May
h2. Pavneet's docs (imported from gisaf's wiki)
807 1 Philippe May
808 159 Selvarani C
809 111 Selvarani C
810 111 Selvarani C
h1. Basic Rules of Map making
811 111 Selvarani C
812 111 Selvarani C
Regardless of the cartographic style or content, most maps have the following common elements.
813 111 Selvarani C
814 111 Selvarani C
815 111 Selvarani C
h2. TITLE
816 111 Selvarani C
817 111 Selvarani C
The title should be in a large font, easily identifiable as the title of the map and should include descriptive text as to the location and purpose of the map. If the map is thematic, the theme should be included in the title. For example: Corn Production in Washington, 1990. The title is usually the largest font size of all lettering on the layout, however, it should not dominate the map graphic itself. The title may or may not be in a box and does not need to be at the top of the page (though it often is). For published materials (e.g., books or articles) the title may be included in a figure caption instead.
818 111 Selvarani C
819 111 Selvarani C
 
820 111 Selvarani C
821 111 Selvarani C
h2. SCALE INDICATOR
822 111 Selvarani C
823 111 Selvarani C
The scale of the map is typically indicated by a graphic bar scale, a representative fraction or a verbal scale. The reader must be able to determine the relationship between a unit of measure on the map and a unit of measure in the real world.
824 111 Selvarani C
825 111 Selvarani C
 
826 111 Selvarani C
827 111 Selvarani C
h2. ORIENTATION
828 111 Selvarani C
829 111 Selvarani C
A map should indicate which way is north (and/or south, east and west). Commonly this is done by a north arrow or compass rose. Orientation may also be shown by graticule or grid marks (e.g. lines of latitude and longitude). By convention north is towards the top of the page (thus some maps do not have north arrows), but the orientation must still be given for a 'proper' map. North does not have to be at the top of the page and a north arrow is essential in maps where it is not.
830 111 Selvarani C
831 111 Selvarani C
h2. BORDER(s)
832 111 Selvarani C
833 111 Selvarani C
A border identifies exactly where the mapped area stops. The border is often the thickest line on the map and should be close to the edges of the mapped area. The distance between the map and the border should be the same on all sides (balanced).
834 111 Selvarani C
835 111 Selvarani C
There can also be a border around the entire map layout (enclosing and grouping the title, legend, text boxes, etc.). 
836 111 Selvarani C
837 111 Selvarani C
Both of these borders are sometimes referred to as a 'neatline.' In addition, there is sometimes a thin additional line just outside of a border (accentuating it and ideally making it more visually appealing) that may also be referred to as a neatline.
838 111 Selvarani C
839 111 Selvarani C
h2. LEGEND
840 111 Selvarani C
841 111 Selvarani C
A legend defines the symbols or colors (including shades of gray and patterns) used on the map. Maps do not need legends if the symbology is so common or simple as to be easily understood by the reader. However, it must be clear what each marker or line type, weight and pattern represents. The legend does not need to be labeled "Legend." The more complicated the symbology on a map the more important the legend becomes.
842 111 Selvarani C
843 111 Selvarani C
h2. MAP CREDITS
844 111 Selvarani C
845 111 Selvarani C
* SOURCE OF DATA (especially on thematic maps)
846 111 Selvarani C
* NAME of the cartographer  
847 111 Selvarani C
* DATE of the map creation/publication 
848 111 Selvarani C
* DATE of the map data 
849 111 Selvarani C
* PROJECTION of the map (especially small-scale maps)
850 111 Selvarani C
851 111 Selvarani C
h2. LOCATOR MAP (INSET)
852 111 Selvarani C
853 111 Selvarani C
A locator map is needed if the area of the map is not easily recognizable or is of large scale. For example, if you map Whatcom County, there should be an inset map of Washington, showing the location of Whatcom County.  Inset DETAIL map(s) may also be used to show an area of the map in greater detail (larger scale).
854 111 Selvarani C
855 111 Selvarani C
h2. EFFECTIVE GRAPHICAL DESIGN
856 111 Selvarani C
857 111 Selvarani C
The layout design is as important as effective sentence structure is to written text. Layout design refers to the planning and decision making processes involved in the visual display of the spatial data. You can achieve balance by rearranging the map elements (north arrow, legend, scale, title, etc.) and changing size of the text, border. etc. The map and map elements should be:
858 111 Selvarani C
859 111 Selvarani C
* Neatly drawn
860 111 Selvarani C
* Appropriately and consistently generalized
861 111 Selvarani C
* Symmetrically balanced (avoid crowding or large blank areas)
862 111 Selvarani C
* Without unnecessary clutter (keep it simple, be wary of 'artistic' details)
863 111 Selvarani C
864 111 Selvarani C
h2. VISUAL HIERARCHY
865 111 Selvarani C
866 111 Selvarani C
A hierarchy of symbology should be used for the lettering, line weights and shading. More important features are typically larger and/or darker, less important/background information should be smaller and/or lighter. At the same time, do not "over weight" or "under weight" features.
867 111 Selvarani C
868 111 Selvarani C
 
869 111 Selvarani C
h2. PURPOSE
870 111 Selvarani C
871 111 Selvarani C
All maps have a purpose which should influence every element of the map and the map layout. A cartographer should be able to clearly articulate the purpose of their map and should keep the audience (who the map is going to be used by) and the client (who the maps is being produced for) in mind.
872 111 Selvarani C
873 111 Selvarani C
h3. NOTE: Any, or all, of the above 'rules' can be (and frequently have been) violated at the discretion of the cartographer IF doing so produces a better map (better serving its purpose and audience).
874 111 Selvarani C
875 111 Selvarani C
*In general, with cartography, less is more (avoid excessive clutter).*
876 1 Philippe May
877 161 Selvarani C
h1. GIS - Survey_Database
878 112 Selvarani C
879 112 Selvarani C
DB- Database
880 112 Selvarani C
C3D- Civil 3D
881 112 Selvarani C
882 112 Selvarani C
h2. How to create Survey database in Civil 3D
883 112 Selvarani C
884 112 Selvarani C
h3. 1. Setting up the Working Folder
885 112 Selvarani C
886 112 Selvarani C
A working folder needs to be created where the survey databases gets stored. In the *Toolbox* > Right click on *Survey databases* > Set the working folder - Save the folder in the desired location. We are setting the Working Folder for the Civil 3D databases in C:
887 112 Selvarani C
888 117 Selvarani C
!https://redmine.auroville.org.in/attachments/download/9783/working%20folder%20image.jpg!
889 116 Selvarani C
890 112 Selvarani C
h3. 2. Creating a database
891 112 Selvarani C
892 112 Selvarani C
Right click on *Survey databases* > *New local Survey database* - Enter the name> click ok. A survey DB is created (It is in bold which signifies it is the current DB under use). Multiple DB's can be created. A survey DB can be opened for editing by right click > *open for edit*. It can be closed by right click > *close for edits*. A survey DB can also be opened as *read-only* by same procedures.
893 112 Selvarani C
894 112 Selvarani C
h3. 3. Setting up the database
895 112 Selvarani C
896 112 Selvarani C
Survey Database(abc) > Right click + Survey Database Settings > Specify co-ordinate Zone > Distance- Metre > Temperature - Celsius > Pressure - Millibars > Distance type - Horizontal > Vertical type - Vertical Distance.
897 112 Selvarani C
898 112 Selvarani C
Then the next step is to create the Networks under which the point data gets uploaded. For example in our case we have - TS and RTK
899 112 Selvarani C
900 112 Selvarani C
h2. Components of Survey Database
901 112 Selvarani C
902 112 Selvarani C
Survey database in civil 3D has the following components when expanded. 
903 112 Selvarani C
# Import Events
904 112 Selvarani C
# Survey Queries
905 112 Selvarani C
# Networks
906 112 Selvarani C
# Network Groups
907 112 Selvarani C
# Figures
908 112 Selvarani C
# Figure Groups
909 112 Selvarani C
# Survey Points
910 112 Selvarani C
# Survey point Groups
911 112 Selvarani C
912 118 Selvarani C
!https://redmine.auroville.org.in/attachments/download/9782/Survey_db.png!
913 163 Selvarani C
914 113 Selvarani C
915 113 Selvarani C
h1. Survey- Field to Finish
916 113 Selvarani C
917 113 Selvarani C
h2. Steps from field work 
918 113 Selvarani C
919 113 Selvarani C
# ?? To be added by Raj and Ram
920 113 Selvarani C
# Copy the .txt files from the equipment- Controller/Total station, to system using a pendrive (Storing the data- "D:"> "AVSM"> "Water Projects"> Respective project folder)> "TS"/"RTK")
921 113 Selvarani C
# Cleaning up the .txt files/removing errors (If any)
922 113 Selvarani C
923 113 Selvarani C
h2. Feeding the survey data into Civil 3D- by Surveyor
924 113 Selvarani C
925 113 Selvarani C
# After cleaning up of files, the files are stored in a desired location, to be used as import events into the Civil 3D file
926 113 Selvarani C
# Open up project in Civil 3D
927 113 Selvarani C
# Importing of Events (explain the step)
928 113 Selvarani C
929 113 Selvarani C
h2. Processing- by Surveyor
930 113 Selvarani C
931 113 Selvarani C
Initial line work is generated from survey points in the drawing. Points for reconciliation (Changing the point codes) are reported, if any. It is important that there is a fair knowledge of Autodesk Civil 3D for this work.
932 113 Selvarani C
933 113 Selvarani C
h2. Post-Processing
934 113 Selvarani C
935 113 Selvarani C
Final linework is generated on top of the processed linework. At this stage, a drawing is cleaned and prepared for sharing. problems like - overlaps in line-work, proper assigning of layers etc. Points for reconciliation are reported, if any. Post-Processing is crucial to generate correct and standardised survey drawings (in .dwg format) keeping in mind that those drawings will be used to generate shape files (.shp format). Therefore, it becomes important to follow a certain workflow based on compatibility of .dwg elements and .shp elements - for example for creating a shapefile of "Polygon" type, the elements in .dwg should be all "Polylines" of "closed" nature. All features should be in zero " 0 " elevation and the shapefile generated should be 3d type. This has been elaborated here < insert > At this stage, a good knowledge of Autodesk Civil 3D and GIS (to a certain extent) is a must.
936 113 Selvarani C
937 113 Selvarani C
h2. Creating shapefiles in AutoCAD
938 113 Selvarani C
939 113 Selvarani C
?? add by pavneet
940 113 Selvarani C
941 113 Selvarani C
h2. Sharing on WebGIS - "GISAF-  +gis.auroville.org.in+" 
942 113 Selvarani C
943 113 Selvarani C
After creating the shp files from AutoCAD, the shp files are ready to be shared on WebGIS platform. Following are the steps to do so.
944 113 Selvarani C
945 113 Selvarani C
946 113 Selvarani C
# *Upload and import Raw survey points to GISAF* - Before uploading and importing the shapefiles, it is necessary to upload and import the corresponding Raw survey points into GISAF.
947 113 Selvarani C
# *Auto-import of point shapefiles* - After the import of point files (RTK and TS), the next step is to Auto-import the points which are point type shp files. Go to Admin> Others> Projects> Select the corresponding project > With selected > Auto- import to GIS Database ( GISAF)
948 113 Selvarani C
# *Upload shp to basket* - Zip together all the types of files obtained after saving a shapefile on the system (shp, prj, dbf etc). Follow the same naming standards when creating this. Upload it to the basket in GISAF- Admin> Basket> Shapefiles> click on relevant project> click on relevant sub-project (if any)> upload file> select the zipped file from system> select the category from directory >save.
949 113 Selvarani C
# *Import to GISAF (Only with Authorization)* - Click on the import arrow.
950 113 Selvarani C
951 113 Selvarani C
h2. Adding new codes
952 113 Selvarani C
953 113 Selvarani C
ADD !!
954 113 Selvarani C
955 113 Selvarani C
h2. Reconciliation of points
956 113 Selvarani C
957 113 Selvarani C
The points are noted with their codes from and codes to reconcile into. ADD!!
958 119 Selvarani C
959 119 Selvarani C
h1. Survey Data Post-Processing
960 119 Selvarani C
961 119 Selvarani C
+Softwares used: Civil 3D, QGIS+
962 119 Selvarani C
963 119 Selvarani C
AutoCAD Civil 3D - C3D, 
964 119 Selvarani C
965 119 Selvarani C
C3D is being used for post processing of survey data. C3D offers a BIM solution for Land Surveyors. Basic knowledge about C3D can be obtained by following tutorials online though a pre-acquired knowledge of Autodesk AutoCAD is a plus point and sufficient enough to start working with C3D.
966 119 Selvarani C
967 119 Selvarani C
QGIS is an Open source GIS.
968 119 Selvarani C
969 119 Selvarani C
Getting Started:
970 119 Selvarani C
+Standards and Workflow for C3D+
971 119 Selvarani C
The layers (Nomenclature of layers is coming from standards- U.S. National CAD Standard Version-AIA) are pre-assigned a geometry type (Point, Line and Polygon) in their layers description (Use layer manager to check). This is done keeping in mind inter-operability with GIS (In shapefiles format), geometry type Shape files are of three kinds- points, lines and polygons.
972 119 Selvarani C
973 119 Selvarani C
+Geometry types-+
974 119 Selvarani C
Based on the description of Layers, if the layer Geometry is specified as 'Point', the raw survey data can be auto imported to webGIS - *GISAF*, no post processing is needed for 'Points'. For lines and polygons, We use Polylines (for 2D lines/Polygons) and 3Dpolylines (for 3D lines/polygons). In case of curved lines in 2D, Polylines command (PLINE/PL) is enough but its not possible to create curved lines in 3D using the 3Dpolylines, in such cases feature lines come into the picture. (*Important:* 2D polylines, splines, ellipses and circles are not exportable to shapefile format, they have to be always converted into Polylines).
975 119 Selvarani C
976 119 Selvarani C
*We are at the moment generating shapefiles (lines and polygons) in 2D (z=0) but keeping the format of files as 3D.
977 119 Selvarani C
978 119 Selvarani C
+Using Feature lines for creating Curved geometries in 3D-+
979 119 Selvarani C
There are a series of steps involved when creating curved lines using feature lines. You can follow two methods depending on the situation :-
980 119 Selvarani C
981 119 Selvarani C
Method 1(For curved objects in 3D using elevation of points):- Create the object using 3DPOLY. Now use the create feature lines from objects and select the objects to be converted to feature lines. You will not notice any difference in the geometry on the screen as such but in properties you will see the object type description of selected object as feature lines.
982 119 Selvarani C
983 119 Selvarani C
Method 2(For curved objects in 3D by draping the lines/curves on a surface):- Create the object using PLINE. You will notice all the lines are straight. Now use the create feature lines from objects and select the objects to be converted to feature lines. You will not notice any difference in the geometry on the screen as such but in properties you will see the object type description of selected object as feature lines.
984 119 Selvarani C
985 119 Selvarani C
Now modify tab is used to convert the straight lines to curved lines. There are many ways to do it and different methods can be adopted in different situations. The easiest and the most commonly used method is by using the smooth command in the modify tab. After using the smooth command, a curve would be visible. This is a curve in 3D. To view it, object viewer option can be used by selecting the object and right clicking.
986 119 Selvarani C
More complex methods have to be studied and each case has to be taken into consideration separatly when using - FITCURVEFEATURE In some cases to obtain a desired curve(Meaning more fragments) more PI's (Point of insertion) need to be added into the feature line. PI are the points where the feature lines gets fragmented when exploded.This step often becomes difficult to manage due to complexities involved in mathematical functions behind creation of feature lines.But it should be explored by all means.
987 119 Selvarani C
Because the feature lines are not exportable to shapefiles, the feature line has to converted back into a 3D polyline. This can be done simply by explolding the object. Once exploded into 3D polylines, you will see that the curve gets fragmented into smaller segments of straight lines. You can repeat the process of creating the 3Dpolyline object into feature line and then smoothening it till you get a desired geometry resembling the curve.
988 119 Selvarani C
Situations of sharing of points with different layers and proceeding with linework in such circumstances-
989 119 Selvarani C
There is always a best way to take the survey points taken in the field by the surveyor. For instance, in case of a road and a curb adjacent to it, the surveyor takes the points only once. It is only during the post processing that the lines are generated (You can create the lines in field while taking the survey points as well, but that is not the most feasible method majorly due to time constraints) and in this case, two line on the same position shall be generated, one under layer of roads and one under layer of curbs. 
990 119 Selvarani C
The line work becomes complex in case of generating 3d lines and 2d lines on the same place. This can arise due to having some layers as 2D (for example building outlines) and some as 3D (paved surfaces- <taking into consideration the future use of the layers. In this case, for the purpose of water management, it becomes essential to have survey of such surfaces as 3D. Survey of building outlines doesn't need to be 3D because the elevation points are not recorded while surveying them due to feasibility).
991 119 Selvarani C
So, in case of generating 3D linework from survey points with no elevation, Feature lines and SURFACES come into picture. The surfaces are generated by an interpolation method in C3D by giving a set of Point Groups. We have selected the triangulation method as the appropriate method due to availability of a dense set of points. 
992 119 Selvarani C
While creating the feature lines from objects (as explained above, check the box "assign elevation" and a dialog box appears to select the surface you want to select). Surfaces will have to be generated prior to using this. (add wiki for generating surface). Surface should be generated including the points of the layer in the point group of it. 
993 119 Selvarani C
994 119 Selvarani C
*+Purpose of Survey+*
995 119 Selvarani C
996 119 Selvarani C
# The survey is conducted for generating a Base Map which is representative of Topography, Infrastructure, Drainage, Buildings etc, with a focus on Water Management in Auroville. Furthermore this survey can be used as base map to build upon more detailed surveys for the various purposes such as Town planning, Land Surveys, Transportation Planning etc.
997 119 Selvarani C
998 119 Selvarani C
999 119 Selvarani C
+Use of Layers+
1000 119 Selvarani C
1001 119 Selvarani C
# *V-BLDG-SHED* (Shed) layer is used when the structure is not entirely enclosed, if At least one side is open (No walls/partition etc). One thumb rule is to answer the question if the building can be locked safely or not. If not, then it is a shed.
1002 119 Selvarani C
# *V-BLDG-HUT* (Hut) is used when the building (For human activity/living space) is single storey with roof made up of perishable material such as keet, straw etc.
1003 119 Selvarani C
# Use of *V-BLDG-ROOF* (Roof) is for buildings with permanent roofs extending out about more than 1 meter from the outline at ground level. It can also be used for complex roof structures for example Matrimandir Petals.
1004 119 Selvarani C
# Use of *V-BLDG-OTLN* (Building outline) is for depicting the outer line of walls of a structure on ground level. This is used when the roof of the building is more or less of the same profile.
1005 119 Selvarani C
# *V-BLDG-RTWL* (Retaining Wall) is for walls with the function of retaining either earth/water and the side of the wall should be visible. In case of very thin walls and no side visible, it comes as *V-BLDG-RWLL* (Retaining wall line). This is depicted as a single line in drawing.
1006 119 Selvarani C
# Use of *V-ROAD-CYCP* is for designated cycle path as a single line.
1007 119 Selvarani C
# Use of *V-ROAD-FPAT* is for pedestrian paths as a single line. It is used when there is a path of very small width. If the need is to represent a path/road which is unpaved in nature, *V-ROAD-UPVD* is used, which is polygon type. To represent a path/road of paved nature *V-ROAD-PAVD* is used, which is also a polygon type.
1008 119 Selvarani C
# *V-WMNG-PIPP* for pipe points, *V-WMNG-PIPL*- for pipe lines.
1009 119 Selvarani C
# *V-BLDG-BMRK* is for unidentified bench marks and *V-BLDG-MHOL* is for unidentified manholes.
1010 119 Selvarani C
# *V-BLDG-PLTF* is for impervious surfaces. for example sand and gravel will not come under this layer. cemented paths etc can come.
1011 119 Selvarani C
# *V-WMNG-POOL* is for water bodies made for leisure activities- like swimming pool etc.
1012 119 Selvarani C
# *V-WMNG-SUMP* is collectively for artificial water retention structures.
1013 119 Selvarani C
# *V-WATR-POND* is for top of natural water retention.
1014 119 Selvarani C
# *V-WATR-PONB* is for bottom of natural water retention.
1015 119 Selvarani C
# *V-WMNG-DRBA* is for bottom of artificial drains.
1016 119 Selvarani C
# *V-WMNG-DRTA* is for top of artificial drains.
1017 119 Selvarani C
# *V-WATR-DRBN* is for bottom of Natural drains.
1018 119 Selvarani C
# *V-WATR-DRTN* is for top of Natural drains.
1019 119 Selvarani C
# *V-WMNG-DRNC* is for drain covers.
1020 1 Philippe May
# *V-WATR-DRAS* is for indicating slope of drainage, these are arrows.
1021 119 Selvarani C
1022 120 Selvarani C
h1. Wells Documentation
1023 120 Selvarani C
1024 120 Selvarani C
Documentation of the wells in Auroville started afresh in September 2017 by Bala working with the *Water Group*. He used a mobile GPS to record co-ordinates and took pictures along with other data related to a well. An effort was made to reconcile data by identification  of the wells with existing data from *Auroville Water Harvest* which ceased to exist around 2007/2009. In some cases the codes on the well on location (sometimes there is a code on the pump, on the casing or on a nearby wall) helped in reconciliation but mostly by spatial mapping using QGIS. Some wells could not be reconciled due to absence of any nearby well in old data. However, we are publishing all the wells we have documented so far.
1025 120 Selvarani C
1026 120 Selvarani C
*Terms of reference*
1027 120 Selvarani C
1028 120 Selvarani C
+Well status+ 
1029 120 Selvarani C
	
1030 120 Selvarani C
# *In use*      Well is equipped with functioning a pump	
1031 120 Selvarani C
# *Not in use*  Well is not equipped with a functioning pump. (Special case- well is equipped with a non functioning pump)
1032 120 Selvarani C
# *Closed*      Well is closed/ sealed/ abandoned, (Historical reference)
1033 120 Selvarani C
1034 1 Philippe May
Note: Wells with pump under repair (temporary measure) at the time of survey are treated as *In use*.
1035 120 Selvarani C
1036 167 Selvarani C
1037 121 Selvarani C
1038 121 Selvarani C
h1. Civil 3D useful commands
1039 121 Selvarani C
1040 121 Selvarani C
1041 121 Selvarani C
h2. Making Feature line to polyline-
1042 121 Selvarani C
1043 121 Selvarani C
Select the feature line> go to elevation editor (under feature line tab> edit elevations)> select all the points in table, give elevation(this will give same elevation to all the points thereby making it possible to retain the curves in the polyline) > explode the feature line (use X enter)
1044 121 Selvarani C
1045 121 Selvarani C
h2. Converting Circle to polyline-
1046 121 Selvarani C
1047 121 Selvarani C
BR > enter > break the circle at two points to obtain a part of circle. 
1048 121 Selvarani C
PEDIT> enter> select the part of leftover circle> J > enter> Close> enter.
1049 121 Selvarani C
1050 121 Selvarani C
h2. Converting 2DPOLYLINE to POLYLINE
1051 121 Selvarani C
1052 121 Selvarani C
Explode the 2D polyline, and use the PEDIT command to convert the segments to polylines. Then join the polylines.
1053 121 Selvarani C
1054 121 Selvarani C
h2. Viewing only the used layer in ACAD-
1055 121 Selvarani C
1056 121 Selvarani C
Set the value of SHOWLAYERUSAGE from 0 to 1
1057 121 Selvarani C
1058 121 Selvarani C
h2. Convert 3d polyline to polyline- 
1059 121 Selvarani C
1060 121 Selvarani C
COVERT3DPOLYS (Change the elevation of the polyline using properties manager to 0 in our case)
1061 121 Selvarani C
1062 121 Selvarani C
h2. Generating contours from a Surface in Civil 3D
1063 121 Selvarani C
1064 121 Selvarani C
# In prospector, Create a point group of points needed to generate the contours from for example- TOPO elevation points or BLDG floor levels. If the point group is already present, then proceed to next step.
1065 121 Selvarani C
# Create a surface in the prospector> when creating the new surface, give the name and styling in the dialog box- styling used by us is "Triangles and surface- 0.1 and 0.5"> Assign the point groups in the surface as created in previous step.
1066 121 Selvarani C
# Go to the surface created in prospector and right click > Edit surface style> Turn on the Major and Minor contour.
1067 121 Selvarani C
# Turn on the layers for Major and Minor contours in Layers Manager (LA > enter)
1068 121 Selvarani C
1069 121 Selvarani C
h2. Shortcuts:
1070 121 Selvarani C
1071 121 Selvarani C
• Properties manager - ctrl + 1
1072 121 Selvarani C
• To view 3D- Select and right click> object Viewer.
1073 121 Selvarani C
• To copy the Line - Copy and select the line enter 
1074 121 Selvarani C
• To convert spline to polyline >Splinedit
1075 121 Selvarani C
• To export shapefile - Mapexport > follow the process.
1076 121 Selvarani C
• To assign coordinate system to the drawing - MAPCSLIBRARY
1077 121 Selvarani C
• To check coordinate system of the drawing- > TOOLSPACE> Settings> Right click on the drawing name > Edit settings
1078 121 Selvarani C
1079 122 Selvarani C
h1. Online references for Civil 3D
1080 122 Selvarani C
1081 123 Selvarani C
*Description Key Sets*
1082 122 Selvarani C
1083 122 Selvarani C
https://www.youtube.com/watch?v=Von8oCwYcTk
1084 122 Selvarani C
1085 124 Selvarani C
*Autodesk civil 3D Geotechnical module- for borehole data*
1086 122 Selvarani C
1087 122 Selvarani C
https://www.youtube.com/watch?v=0Be9kShBou0
1088 122 Selvarani C
1089 123 Selvarani C
*Civil 3D for Surveyors*
1090 122 Selvarani C
1091 122 Selvarani C
https://www.youtube.com/playlist?list=PL1EzH8XlwSxuxCMdeLvzqfgsAlmeeHMv2
1092 122 Selvarani C
1093 123 Selvarani C
*Civil 3D: Survey - Survey Database*
1094 1 Philippe May
1095 122 Selvarani C
https://www.youtube.com/watch?v=VPWdAfYJt5Y 
1096 123 Selvarani C
1097 169 Selvarani C
1098 125 Selvarani C
1099 125 Selvarani C
h1. Connections in QGIS- Using browser panel and Add postGIS
1100 125 Selvarani C
1101 125 Selvarani C
Working using QGIS as interface using PostGIS connections.
1102 125 Selvarani C
1103 125 Selvarani C
* Server: @gisdb.csr.av@
1104 125 Selvarani C
1105 125 Selvarani C
* Database: @avgis@
1106 125 Selvarani C
1107 125 Selvarani C
* Leave @Service@ empty
1108 125 Selvarani C
1109 125 Selvarani C
1110 125 Selvarani C
h2. Adding tables (With geometries) in form of shape files from database using Browser panel-
1111 125 Selvarani C
1112 125 Selvarani C
Go to QGIS> view> Panels> Browser panel> PostGIS> Expand the connection> give credentials> add the desired file by double clicking on it.
1113 125 Selvarani C
1114 125 Selvarani C
1115 125 Selvarani C
h2. Adding tables (For non-geometry type) using PostGIS connections-
1116 125 Selvarani C
1117 125 Selvarani C
Add PostGIS Layers > Give credentials > select "Also list tables with no geometry" > expand public > click on the desired table> add.
1118 125 Selvarani C
1119 125 Selvarani C
h2. Joining tables
1120 125 Selvarani C
1121 125 Selvarani C
Right click/double click on file> go to Join > perform the desired joins - add/subtract the joins. 
1122 125 Selvarani C
1123 125 Selvarani C
1124 22 Pavneet Kaur
[[Reconcilation of Raw survey data using pgAdmin]]
1125 126 Selvarani C
1126 126 Selvarani C
*Reconcilation of Raw survey data using pgAdmin*
1127 126 Selvarani C
1128 126 Selvarani C
+For changing the layer codes, to be done using Pgadmin.+
1129 126 Selvarani C
1130 126 Selvarani C
h3.  Enter Schema> AVSM RAW Survey > Tables > select the table> Right click - View edit data > All rows. Apply filter to the original id and note the Database id's to be reconciled.
1131 126 Selvarani C
1132 126 Selvarani C
h3.  GISAf Admin> Other > Reconciliation. Create > Add the database (point) id and give the new target corresponding to the database id to be changed. (Use the Layer name in target not Raw layer name).
1133 126 Selvarani C
1134 126 Selvarani C
h3.  Other > Project > select the project > with selected> reconcile RAW survey points.
1135 126 Selvarani C
1136 26 Pavneet Kaur
[[importing RAW data to GISAF]]
1137 127 Selvarani C
1138 127 Selvarani C
h1. Importing point data (TS and RTK) to GISAF 
1139 127 Selvarani C
1140 127 Selvarani C
# *Gisaf Admin > Basket > Survey data > "Project" > "surveyor" > TS/ RTK > upload > import.*
1141 127 Selvarani C
# *Auto import of Raw points data* (changes from Raw to Shapefiles, the point files)
1142 127 Selvarani C
  In GISAF Admin > Other> Project > select the project for which you want to import the data > with selected > Auto import to GIS...
1143 127 Selvarani C
1144 34 Pavneet Kaur
[[Editing Z value of features in Shapefiles in QGIS]]
1145 128 Selvarani C
1146 128 Selvarani C
h1. Editing Z value of features in Shapefiles in QGIS
1147 128 Selvarani C
1148 128 Selvarani C
h2. using vertex editor tool - https://www.youtube.com/watch?v=8V8i1AtcA74&t=256s
1149 128 Selvarani C
1150 36 Pavneet Kaur
[[Miscellaneous- Civil 3D]]
1151 129 Selvarani C
1152 129 Selvarani C
h1. Miscellaneous- Civil 3D
1153 129 Selvarani C
1154 129 Selvarani C
*Autodesk civil 3D Geotechnical module- for borehole data :- To analyse borehole data, To make profiles and calculate volumes*
1155 129 Selvarani C
1156 129 Selvarani C
https://www.youtube.com/watch?v=0Be9kShBou0
1157 129 Selvarani C
Reference videos
1158 129 Selvarani C
Exploring:
1159 129 Selvarani C
https://www.youtube.com/watch?v=gr-ISPzLcU0
1160 129 Selvarani C
Modelling:
1161 129 Selvarani C
https://www.youtube.com/watch?v=Pz0-HOoiBrs
1162 129 Selvarani C
solids
1163 129 Selvarani C
https://www.youtube.com/watch?v=mQ0Yeh6tZA8
1164 129 Selvarani C
https://www.youtube.com/watch?v=4eMsUiYBhuE
1165 129 Selvarani C
1166 129 Selvarani C
1167 129 Selvarani C
*CIVIL 3D Survey*
1168 129 Selvarani C
Getting started- Always open a new drawing with a template. 
1169 129 Selvarani C
1170 129 Selvarani C
*Description key sets*- (till 18:30) https://www.youtube.com/watch?v=mmwkkRyBkS0
1171 129 Selvarani C
As when the points come into the drawing, they are going to be filtered with description key sets and the description key sets will assign properties to the points.
1172 129 Selvarani C
Tool space > settings > points > description key sets
1173 129 Selvarani C
“Description key sets name”> edit key- to view the points in a list: 
1174 129 Selvarani C
Automatic linework (18:30- till end)
1175 129 Selvarani C
Survey > Linework code sets- for automatic linework
1176 129 Selvarani C
1177 129 Selvarani C
*Using Master view*
1178 129 Selvarani C
Uses-
1179 129 Selvarani C
copy styles from one drawing to other
1180 129 Selvarani C
copy drawing data from one drawing to other (data referencing)
1181 129 Selvarani C
1182 129 Selvarani C
*Civil 3d surface model*- from points- break lines
1183 129 Selvarani C
https://www.youtube.com/watch?v=wC40rdmDEMo
1184 129 Selvarani C
moving for example- point groups under point groups- 
1185 129 Selvarani C
1186 129 Selvarani C
*Civil 3D Planning and Analysis*
1187 129 Selvarani C
Workspace: Planning and Analysis
1188 129 Selvarani C
1. Working with Object data (GIS Attributes)
1189 129 Selvarani C
Map Setup> Define object data> New table> Define new object data table start defining fields
1190 129 Selvarani C
1191 129 Selvarani C
1192 40 Pavneet Kaur
[[Documentation- Rain Gauge]]
1193 130 Selvarani C
1194 130 Selvarani C
h1. Documentation- Rain Gauge
1195 130 Selvarani C
1196 130 Selvarani C
h2. Manual Rain Gauge
1197 130 Selvarani C
1198 130 Selvarani C
*Why it is important to comply to standards?*
1199 130 Selvarani C
1200 130 Selvarani C
1. Consistency for comparing rainfall data in different places within Auroville since rainfall varies in different parts of Auroville.
1201 130 Selvarani C
1202 130 Selvarani C
2. Making it possible to use for any kind of scientific analysis by contribution to the primary data in sustainable water management for Auroville. 
1203 130 Selvarani C
1204 130 Selvarani C
*Proposed standards and ethics:*
1205 130 Selvarani C
1206 130 Selvarani C
1. The time of taking the reading - 08:30 am. 
1207 130 Selvarani C
The rain is recorded over a period of 24 hours- 8:30 of previous day till 8:30 of the present day, and the date is put as the present day. For example, if the rain is measured at 8:30 am on 10 dec 2017, then it is logged in as on 10 dec 2017.
1208 130 Selvarani C
1209 130 Selvarani C
2. Manual rain gauge typically used- green cylindrical and a collection jar.  
1210 130 Selvarani C
  
1211 130 Selvarani C
3. Measuring jar: 10 mm corresponding jar
1212 130 Selvarani C
1213 130 Selvarani C
 
1214 130 Selvarani C
Area of the rim of manual rain gauge= 200 square cm
1215 130 Selvarani C
Diameter of rim= approximately 16 cm
1216 130 Selvarani C
*NOTE:* - If another kind of rain gauge is used, it should be used with the corresponding measuring jar as type of jar and calibration on jar depends on the area of rim. Any cases of different rain gauge other than above should be reported before use for recording purposes.
1217 130 Selvarani C
1218 130 Selvarani C
4. The minimum record-able unit is 0.2 mm (Least measure on the 10 mm cylinder). T is marked for below 1 mm.
1219 130 Selvarani C
1220 130 Selvarani C
5. The reading should be preferably submitted in an excel sheet (the data sheet will be provided). The frequency of sending data can be daily or weekly.
1221 130 Selvarani C
6. The rain gauge should be kept in a safe and open to sky area clear from tree cover or objects in surrounding so that direct rain falls into it and not from trees/ objects etc. Preferably roof top or a higher place on ground.
1222 130 Selvarani C
7. A check on the rain gauge should be kept when there is long period of gap in rainy days to make sure it was emptied out and there is no blockage. 
1223 130 Selvarani C
1224 130 Selvarani C
8. In case of doubt in reading or incorrect measure, "incorrect measure" should be marked
1225 130 Selvarani C
1226 130 Selvarani C
1227 130 Selvarani C
*Ethics*
1228 130 Selvarani C
1. Readings should not be missed. Time to time check on rain gauge is required during long periods of no rain (especially when the rain season is approaching). 
1229 130 Selvarani C
This is important because firstly, if it rained in the night and it wasn’t recorded, reading for the day is lost; Secondly, if the jar was not emptied, it gives incorrect subsequent reading. 
1230 130 Selvarani C
1231 130 Selvarani C
2.  If the person is not going to be available temporarily where the rain gauge is kept, the responsibility should be passed on to someone they can rely on after explaining the standards. 
1232 130 Selvarani C
1233 130 Selvarani C
*Important-*
1234 130 Selvarani C
Data won’t be published in case of non-compliance to standards. This is to maintain the sanctity of proper scientific data collection and to keep it reliable for sharing by publishing. Discussions regarding standards are welcomed.
1235 130 Selvarani C
CSR Geomatics Team is placed on first floor in CSR, Auroshilpam.
1236 130 Selvarani C
1237 130 Selvarani C
*AV rain data publication*: (http://gis.auroville.org.in/measures/raingauge_av).
1238 130 Selvarani C
1239 130 Selvarani C
Adding Rain gauge to our web portal (GISAF), following information is needed:
1240 130 Selvarani C
1. GPS co-ordinates of location of placing the rain gauge (can be obtained using mobile phones easily)
1241 130 Selvarani C
2. Name, place of residence & contact number- Mobile & Landline
1242 130 Selvarani C
1243 130 Selvarani C
h2. Automatic Rain Gauge
1244 130 Selvarani C
1245 130 Selvarani C
There is a possibility of publishing rain data coming from Automatic rain gauges like (id 15, auro orchard) and Weather stations. 
1246 130 Selvarani C
For Automatic rain gauges, we can upload the files coming from the rain gauge directly into GISAF.
1247 130 Selvarani C
1248 130 Selvarani C
1249 130 Selvarani C
_Thank you for contribution towards a sustainable management of Water in Auroville through Data collection._
1250 130 Selvarani C
1251 130 Selvarani C
1252 40 Pavneet Kaur
[[Documentation- Wells Monitoring (Piezometer by Bala)]]
1253 131 Selvarani C
1254 131 Selvarani C
h1. Documentation- Wells Monitoring (Manual and Automatic / Piezometer by Bala)
1255 131 Selvarani C
1256 131 Selvarani C
h2. Manual- Using tape with a sensor
1257 131 Selvarani C
1258 131 Selvarani C
Timings: The person assigned the job of monitoring collects the readings in three slots.
1259 131 Selvarani C
1260 131 Selvarani C
1.	Between 6 am to 7 am till about 9 am to 10 am.
1261 131 Selvarani C
2.	Between 11 am and 1 pm
1262 131 Selvarani C
3.	Between 2 pm and 5 pm
1263 131 Selvarani C
1264 131 Selvarani C
Things required:
1265 131 Selvarani C
•	Notebook
1266 131 Selvarani C
•	Pen
1267 131 Selvarani C
•	Measuring tape
1268 131 Selvarani C
•	Vehicle for movement
1269 131 Selvarani C
1270 131 Selvarani C
1271 131 Selvarani C
*Ethics*- 
1272 131 Selvarani C
•	The monitoring should happen in coordination with the community members/ care taker (assigned by the person in charge from within the community). There should be a clear communication from monitor side as to which days the monitoring of a well has to happen and at around what time so that it can be made sure that the pump is not turned on before monitoring. In case a pump was turned on in a well, the monitor should have a gap of about 5 hours on the same day before going again for monitoring.
1273 131 Selvarani C
•	The monitor should have contact numbers of the person in charge/ care taker for any communication.
1274 131 Selvarani C
•	If the monitoring is stopped for any reason at any point, the monitor should communicate the same to the person concerned.
1275 131 Selvarani C
•	The monitor shall take responsibility to inform any kind of changes in a well in terms of its functioning etc.
1276 131 Selvarani C
•	The monitor should report to the CSR Geomatics Team who has responsibility to publish data.
1277 131 Selvarani C
•	The monitor published the data to the website and works with the geomatics team.
1278 131 Selvarani C
1279 131 Selvarani C
1280 131 Selvarani C
*Definitions*:
1281 131 Selvarani C
A master file is maintained with the records of the wells from the field. It is updated when a new well is located. The following set of information are filled out in the Masterfile. 
1282 131 Selvarani C
1283 131 Selvarani C
*Well type*
1284 131 Selvarani C
1.	Open
1285 131 Selvarani C
2.	Borewell
1286 131 Selvarani C
3.	Dug cum borewell
1287 131 Selvarani C
1288 131 Selvarani C
*Well status*
1289 131 Selvarani C
1.	In use
1290 131 Selvarani C
2.	Not in use
1291 131 Selvarani C
3.	Closed
1292 131 Selvarani C
1293 131 Selvarani C
*Pump Status*
1294 131 Selvarani C
1.	Functioning
1295 131 Selvarani C
2.	Not functioning
1296 131 Selvarani C
1297 131 Selvarani C
*Pump Automation*
1298 131 Selvarani C
1.	Manual 
1299 131 Selvarani C
2.	Automatic
1300 131 Selvarani C
1301 131 Selvarani C
* Non-accessibility factors*
1302 131 Selvarani C
1.	Dogs
1303 131 Selvarani C
2.	Heavy slab
1304 131 Selvarani C
3.	Narrow casing
1305 131 Selvarani C
4.	enclosed/locked
1306 131 Selvarani C
5.	permission
1307 131 Selvarani C
1308 131 Selvarani C
*Data Matching Accuracy (reconnecting with harvest wells data)*
1309 131 Selvarani C
1.	High
1310 131 Selvarani C
2.	Medium
1311 131 Selvarani C
3.	Low
1312 131 Selvarani C
1313 131 Selvarani C
*Well coordinates*
1314 131 Selvarani C
For a new well, coordinates of the well position are taken on a mobile GPS.
1315 131 Selvarani C
Person In charge
1316 131 Selvarani C
For communication purpose, contact the person in charge as recorded in directory.
1317 131 Selvarani C
1318 131 Selvarani C
h2. Automatic- Piezometer (by bala, to be edited) 
1319 131 Selvarani C
1320 131 Selvarani C
The calibrations and setting are already done by Azha
1321 131 Selvarani C
The piezometer is taken to the site of unused well. First the depth and Water level  is checked manually using water meter
1322 131 Selvarani C
Then the sensor of the piezometer is inserted into the well 1m above the bottom of the well, so the sensor does not get affected from mud or water inside
1323 131 Selvarani C
The sensor sends the data to an electronic board which translates the signal into the proper output, and then it sends the signal to the transmitting device above, on the ground surface. The transmitter then sends it to the receiver at CSR, the signal is then routed to Talam office through Interneet, enters the software and the reading is processed.
1324 131 Selvarani C
The piezometer sometimes does not have proper signal and so it would not be able to send it
1325 131 Selvarani C
1326 131 Selvarani C
1327 42 Pavneet Kaur
[[Documentation- Flow meter, by Bala]]
1328 132 Selvarani C
1329 132 Selvarani C
h1. Documentation- Flow meter by Bala
1330 132 Selvarani C
1331 132 Selvarani C
Flow meter – for checking the flow of water in pipe
1332 132 Selvarani C
1333 132 Selvarani C
CSR bought ultrasonic flow meter from Chennai and the company (company name? )trained Bala and Vijai(CSR) on how to set it up.
1334 132 Selvarani C
we tested in the west water system pump in csr. then we test in many more place .
1335 132 Selvarani C
 ami, aurodam and buddha garden borewell pumps . we got request from the water service 
1336 132 Selvarani C
Cross check they flow meters .
1337 132 Selvarani C
 so before that  we want to know how our meter works. 
1338 132 Selvarani C
so we tested in our tank . the pump pumping from our sump tank so the flow will be stranded.  when you pump in the bore well  the flow goes up and down. 
1339 132 Selvarani C
we tested two times in  15 minutes.and one time 30 minutes,so we got variation between this three. 
1340 132 Selvarani C
we find 1,5 % error but the flow meret they said  1 % error only.
1341 132 Selvarani C
we cross checked the flow meter of the water service we find some error  in they meter also. 
1342 132 Selvarani C
we tried they bore well flowmeter and they sump tank flow meter also.and we find some error also in they meters.  
1343 132 Selvarani C
then we find some error in (AVWS) meter also.
1344 132 Selvarani C
It is used to check the flow of water in pipe
1345 132 Selvarani C
tools used 
1346 132 Selvarani C
First the outer diameter of the pipe has to be entered in the device. It is measured using Vernier caliper
1347 132 Selvarani C
Then the thickness of the pipe is set which is also measured using vernier
1348 132 Selvarani C
Also the device asks for the material of the pipe. If the pipe material is know it can be set and if it not know then there is an option which is other (mostly PVC, HDPE, and  iron) 
1349 132 Selvarani C
After entering these details the device gives the spacing for the sensors. There are 2 sensors up and down which has to be fixed accordingly
1350 132 Selvarani C
When the sensors are fixed the motor is turned on and the sensors send reading to the display device
1351 132 Selvarani C
This is noted down once every minute and taken for 15 to 30 minutes. Then the average of this is determined. This is done because of the variation in the flow. This gives the flow rate in 1 hour.
1352 132 Selvarani C
This was done in different places to check to flow rate
1353 132 Selvarani C
1354 44 Pavneet Kaur
[[Documentation- DST- Vegetation Indexing]]
1355 133 Selvarani C
1356 133 Selvarani C
h1. Documentation- DST- Vegetation Indexing
1357 133 Selvarani C
1358 133 Selvarani C
h2. Steps for Dzetsaka Classification tool for Vegetation indexing in QGIS
1359 133 Selvarani C
1360 133 Selvarani C
 
1361 133 Selvarani C
 1. Install the plugin Dzetsaka classfication tool.
1362 133 Selvarani C
 2. Open the Raster from the Survey.
1363 133 Selvarani C
 3. Create a polygon shapefile for index sampling. Mark polygons and give the ID's (1,2,3 for Tree, grass, bare land etc) Cover the variations in samples as much as possible. 
1364 133 Selvarani C
     More the samples, better the indexing.
1365 133 Selvarani C
 4. Apply Dzetsaka Classification tool, Select the base raster and the sample- index polygon shapefile created in step 3.
1366 133 Selvarani C
 5. The result is a Raster with DN numbers specified in the Shapefile in step 3.
1367 133 Selvarani C
 6. Apply the Sieve raster command (Raster> Analysis> Sieve)- Try different threshold numbers and view the results till the noise is removed from the Raster.
1368 133 Selvarani C
 7. Polygonise the Raster to Vector (From processing)
1369 133 Selvarani C
 8. Run the v.generalise tool on the shapefile. This tool removes the pixelated boundaries of the polygons in the Vector.
1370 133 Selvarani C
1371 43 Pavneet Kaur
[[Documentation- DST- Interpolation]]
1372 134 Selvarani C
1373 134 Selvarani C
h1. Documentation- DST- Interpolation (Processing toolbox)
1374 134 Selvarani C
1375 134 Selvarani C
The following 4 tools have been mostly explored and the results were compared. The ones used for quick analysis are 1. Cubic Spline and 2. V.surf.spline . The rest of the tools are for further exploration and used depending on the need of the project. In some tools, the elevation values of points should be stored in the attribute table (Using field calculator and giving command - *Z($Geometry)* )
1376 134 Selvarani C
1377 134 Selvarani C
# Interpolate (Cubic spline) - SAGA
1378 134 Selvarani C
# V.surf.bspline - GRASS. Parameters to set - cell size. Set this parameter above 0.001 ( 0.00001, 0.000001 etc) and check the results.
1379 134 Selvarani C
# V.surf.rst - GRASS
1380 134 Selvarani C
# Krigging - SAGA
1381 134 Selvarani C
1382 40 Pavneet Kaur
[[Documentation- DST- Survey- Office workflow]]
1383 41 Pavneet Kaur
[[From CAD to GIS, by Giulio]]
1384 135 Selvarani C
1385 135 Selvarani C
h1. From CAD to GIS by Giulio
1386 135 Selvarani C
1387 135 Selvarani C
*FEATURES CREATION*
1388 135 Selvarani C
1.	Assign a CRS to the drawing (TM-AUSPOS) (MAPCSLIBRARY command)
1389 135 Selvarani C
2.	Create features in CAD (Points, lines, polygons)
1390 135 Selvarani C
3.	Export shapefile (a) from CAD (Output > DWG to SDF) (Convert to LL84 – 3D)
1391 135 Selvarani C
1392 135 Selvarani C
*FEATURES IMPORT INTO DB FIRST TIME*
1393 135 Selvarani C
4.	Create zip file of the shapefile
1394 135 Selvarani C
5.	Upload into the GISAF Shapefiles Basket
1395 135 Selvarani C
6.	Import the shapefile into DB
1396 135 Selvarani C
7.	Save the shapefile on Local Machine
1397 135 Selvarani C
1398 135 Selvarani C
*FEATURES IMPORT INTO DB EVERYTIME*
1399 135 Selvarani C
8.	Combine the new features to corresponding last shape files (Insert the process here).
1400 135 Selvarani C
9.	Follow step 4-8 again
1401 135 Selvarani C
1402 135 Selvarani C
*FEATURES EDITING IN QGIS*
1403 135 Selvarani C
10.	Open the table in QGis
1404 135 Selvarani C
11.	Save as a shapefile (b) in TM AUSPOS CRS
1405 135 Selvarani C
12.	In CAD, open a new drawing and assign AUSPOS CRS
1406 135 Selvarani C
13.	Import the shapefile (b) (MapImport) with Object Data (Data tab > Create Object Data > OK), tick “Import polygons as closed polylines”, then press OK
1407 135 Selvarani C
14.	Edit features
1408 135 Selvarani C
15.	Change workspace into “Planning and analysis”
1409 135 Selvarani C
16.	Export shapefile (a) from CAD (Output > DWG to SDF) with ONLY the id selected (Data Tab > Select Attributes > Object Data > Filename > id) (Convert to LL84 – 3D)
1410 135 Selvarani C
1411 135 Selvarani C
*FEATURES IMPORT INTO DB*
1412 135 Selvarani C
17.	Create zip file of the shapefile
1413 135 Selvarani C
18.	Upload into the GISAF Shapefiles Basket
1414 135 Selvarani C
19.	Import the shapefile into DB
1415 135 Selvarani C
20.	Delete the shapefile from Local Machine
1416 135 Selvarani C
1417 135 Selvarani C
1418 52 Giulio Di Anastasio
[[QGIS- Miscellaneous]]
1419 1 Philippe May
1420 136 Selvarani C
h1. QGIS- Miscellaneous
1421 136 Selvarani C
1422 136 Selvarani C
*QGIS Introduction:*
1423 136 Selvarani C
https://www.birdseyeviewgis.com/blog/2018/2/22/my-favorite-features-of-qgis-30to-date
1424 136 Selvarani C
3D visualization of raster DEM- https://www.youtube.com/watch?v=2KrCsbP0kUs
1425 136 Selvarani C
1426 136 Selvarani C
*Spatial query:* 
1427 136 Selvarani C
Spatial Query is selection of features that satisfies a certain condition which relates to other features in a space.
1428 136 Selvarani C
Using plugin- Spatial query
1429 136 Selvarani C
http://www.geodose.com/2018/03/spatial-query-in-qgis-3.html
1430 136 Selvarani C
Labelling:
1431 136 Selvarani C
Labelling with more than one field names and in different lines
1432 136 Selvarani C
|| '\n' ||
1433 136 Selvarani C
1434 136 Selvarani C
1435 136 Selvarani C
1436 136 Selvarani C
*Hierarchy of extensions*
1437 136 Selvarani C
1438 136 Selvarani C
File levels and their uses.
1439 136 Selvarani C
https://nathanw.net/2014/03/22/all-the-q-files/
1440 136 Selvarani C
1441 136 Selvarani C
+The Project file (.qgs)+
1442 136 Selvarani C
It contains: Layer source pointer + Style information + Composers + a whole heap of other stuff
1443 136 Selvarani C
1444 136 Selvarani C
+The Layer Definition file (.qlr)+
1445 136 Selvarani C
It contains: Layer source pointer + Style information
1446 136 Selvarani C
1447 136 Selvarani C
+The QML file (.qml)+
1448 136 Selvarani C
It contains: Style information
1449 136 Selvarani C
1450 59 Giulio Di Anastasio
h2. Giulio's documentation
1451 52 Giulio Di Anastasio
1452 65 Giulio Di Anastasio
[[Documentation - Reconciliation of points using Gisaf]]
1453 137 Selvarani C
1454 137 Selvarani C
h1. Documentation - Reconciliation of points using Gisaf
1455 137 Selvarani C
1456 137 Selvarani C
Reconciliation of points is a procedure used when a point is stored in a wrong table, because its category was either wrongly recorded in the field by the surveyor, or it has ben reviewed later by the surveyor or the data validator and found to be wrong.
1457 137 Selvarani C
1458 137 Selvarani C
h2. Definition: Raw points are all points coming from the field survey. Raw points can be points referring to a Point feature (e.g. trees, or elevation points, or floor level), or points measured in the field to draw a line (e.g. the vertices of a fence) or a polygon (e.g. the corners of a building outline).
1459 137 Selvarani C
1460 137 Selvarani C
In the overall workflow, *+reconciliation affects only raw points+*. It takes place after the field textfile is uploaded into the basket and its points imported (raw points stored in the raw survey tables). Here you can visualize the Workflow diagram: [[Survey_data]].
1461 137 Selvarani C
1462 137 Selvarani C
1463 137 Selvarani C
If a raw point refers to a point feature, reconciling it means moving it to another category/table meant for point features, not for lines/polygons. So, *+only raw points of point features can be reconciled+*. Raw points pertaining to line features and/or polygon features cannot be reconciled, so these raw points will remain in their original wrong table.
1464 137 Selvarani C
1465 137 Selvarani C
1466 137 Selvarani C
PLEASE NOTE: 
1467 137 Selvarani C
The attributes of a line/polygon, which are derived from their defining raw points, will not be modified by any reconciliation, because *+only raw points of point features can be reconciled+*.
1468 137 Selvarani C
1469 137 Selvarani C
h2. How to perform Reconciliation
1470 137 Selvarani C
1471 137 Selvarani C
To perform reconciliation of points: Login to Gisaf -> click on the G icon on the upper-left corner of the website page -> Manage -> Reconciliation by orig.ID
1472 137 Selvarani C
1473 137 Selvarani C
On the right end side of the screen, clicking on the field "Destination" a list of all categories will appear: these categories refer not to the Raw survey tables, but to the V_ tables of the database (points, lines, polygons).
1474 137 Selvarani C
1475 137 Selvarani C
Under it, in the field "Original ID", the original point number of the point to be reconciled is to be entered.
1476 137 Selvarani C
1477 137 Selvarani C
Clicking on the "Search points" button, the result shows the database unique id of the point, its survey category, its survey date, its geometry type (point, line, polygon), and the Project the point belongs to. In case of multiple points with the same original id (in case of different Projects, the field number of points might be not unique if the numbering of points in the field has restarted) all points having that original id are displayed: thanks to their date or Project or type, it is easy to identify the correct point to be reconciled.
1478 137 Selvarani C
1479 137 Selvarani C
Once the point to be reconciled is identified, clicking on the button "Reconcile" will run the reconciliation, and a message will appear stating that it has been done successfully. An error message can appear if a reconciliation of a raw point of a line/polygon feature has been attempted: this type of points cannot in fact be reconciled.
1480 137 Selvarani C
1481 137 Selvarani C
Once a raw point has been reconciled, *+it cannot be reconciled again+*. In case a raw point has been wrongly reconciled, it cannot be reconciled again through the above procedure, but it has to be reconciled manually through QGis or pgadmin software.
1482 137 Selvarani C
1483 1 Philippe May
[[Documentation - Status and Status Changes]]
1484 138 Selvarani C
1485 138 Selvarani C
h1. Documentation - Status and Status Changes
1486 138 Selvarani C
1487 138 Selvarani C
Status have been created to keep track og changes in surveyed features.
1488 138 Selvarani C
It is an additional (though provided for in AIA standards), single digit value, at the end of the Gisaf Category/Cad layer name.
1489 138 Selvarani C
Each Status need to have a corresponding CAD layer/Gisaf Category, with a short code associatedto it, so that field entries can be done easily.
1490 138 Selvarani C
1491 138 Selvarani C
Status have been defined as follows:
1492 138 Selvarani C
N - New Work
1493 138 Selvarani C
E - Existing o remain
1494 138 Selvarani C
D - Existing to demolish, Demolished or Changed
1495 138 Selvarani C
F- Future work, Proposed feature
1496 138 Selvarani C
T - Temporary work
1497 138 Selvarani C
M - Item to be moved
1498 138 Selvarani C
X - Not in contract
1499 138 Selvarani C
1500 138 Selvarani C
1501 138 Selvarani C
By default, Status is defined as E (Existing). Status can anyhow be changed later using gAdmin, or QGis (through the PostGis connection).
1502 138 Selvarani C
It needs to be done manually, one feature (point, line, polygon) at a time.
1503 138 Selvarani C
1504 138 Selvarani C
1505 138 Selvarani C
In the future Status changes might be incorporated in the Admin panel.
1506 138 Selvarani C
1507 67 Giulio Di Anastasio
[[Documentation - Tags retained after re-import of same geometry]]
1508 67 Giulio Di Anastasio
1509 139 Selvarani C
h1. Documentation - Tags retained after re-import of same geometry
1510 139 Selvarani C
1511 139 Selvarani C
Documentation - Tags retained after re-import of same geometry
1512 139 Selvarani C
The linework for infrastructure survey carried out by Eric Chacra in May 2020 was originally imported with a problem of ambiguity in the "Accuracy" table and in the "Accuracy" table.
1513 139 Selvarani C
The result was that lines did not inherit the attributes survey date, accuracy, equipment, surveyor.
1514 139 Selvarani C
Nevertheless lines were displayed on the Gisaf map, without these attributes, and tags were given to some of these lines.
1515 139 Selvarani C
1516 139 Selvarani C
The values for the two tables ("Accuracy", "Accuracy") have been corrected, ambiguity resolved.
1517 139 Selvarani C
The lines in the layers V-ELEC-UGND------E and V-COMM-CABL------E have been then reimported, the attributes have been properly assigned, and the tags have been retained.
1518 139 Selvarani C
1519 139 Selvarani C
4 August 2020
1520 139 Selvarani C
1521 67 Giulio Di Anastasio
h1. Access to data
1522 67 Giulio Di Anastasio
1523 67 Giulio Di Anastasio
h2. Connection to server directly from CSR
1524 67 Giulio Di Anastasio
1525 67 Giulio Di Anastasio
To connect to the server directly without going through Aurinoco server, the correct url is
1526 67 Giulio Di Anastasio
http://gis.csr.av
1527 67 Giulio Di Anastasio
1528 67 Giulio Di Anastasio
h2. Connection to Gisaf via QGis through WFS / OGC API
1529 67 Giulio Di Anastasio
1530 67 Giulio Di Anastasio
This works only on QGis from version 3.14.15 onward
1531 67 Giulio Di Anastasio
1532 67 Giulio Di Anastasio
In the browser, click on WFS/OGC API, then right-click to create a new connection
1533 67 Giulio Di Anastasio
Give a name (e.g. OGC API Qgis Gisaf)
1534 67 Giulio Di Anastasio
Give the url https://gis.auroville.org.in/ogcapi
1535 67 Giulio Di Anastasio
1536 67 Giulio Di Anastasio
Under the WFS Options box, on Version dropdown, the default option "Maximum" works just fine
1537 67 Giulio Di Anastasio
Click on OK
1538 67 Giulio Di Anastasio
The list of layers will appear in the Browser under WFS/OGC API.
1539 68 Giulio Di Anastasio
1540 68 Giulio Di Anastasio
1541 68 Giulio Di Anastasio
h1. How to create a new projection in QGis
1542 68 Giulio Di Anastasio
1543 68 Giulio Di Anastasio
To create a new projection in QGis, go to menu "Settings", and click on "Custom Projections".
1544 68 Giulio Di Anastasio
A pop-up window appears with a list of all projections defined in QGis projects used by the user so far.
1545 68 Giulio Di Anastasio
Click on the green "+" sign on the right top part of the window to create a new projection.
1546 68 Giulio Di Anastasio
In the "Name" box, type "TM CSRAUSPOS SF1" (which means TM = Transverse Mercator projection; CSRAUSPOS = theparameters for this projection are derived from the processing of DGPS raw data by AUSPOS -  Online GPS Processing Service - https://www.ga.gov.au/scientific-topics/positioning-navigation/geodesy/auspos; SF1 = Scale Factor is 1).
1547 68 Giulio Di Anastasio
In the "Format" dropdown list, select "Proj String (legacy - Not Recommended)"
1548 68 Giulio Di Anastasio
In the "Parameters" box, paste the following "+proj=tmerc +lat_0=12.01605433+lon_0=79.80998934 +k=1 +x_0=370455.630 +y_0=1328608.994 +ellps=WGS84+towgs84=0,0,0,0,0,0,0 +units=m +no_defs".
1549 68 Giulio Di Anastasio
1550 68 Giulio Di Anastasio
Finally, click on OK.
1551 68 Giulio Di Anastasio
1552 68 Giulio Di Anastasio
In a more explicit  way, the parameters mean the following:
1553 68 Giulio Di Anastasio
Map Projection:        TransverseMercator (TM)
1554 68 Giulio Di Anastasio
False Easting:          370455.6300
1555 68 Giulio Di Anastasio
False Northing:        1328608.9940
1556 68 Giulio Di Anastasio
Latitude of Origin:    12°00'57.79560" (DMS)   12.01605433 (DD)
1557 68 Giulio Di Anastasio
Central Meridian:     79°48'35.96164"  (DMS)   79.80998934 (DD)
1558 68 Giulio Di Anastasio
Scale Factor:             1.00000000
1559 68 Giulio Di Anastasio
Zone Width:              6.0°
1560 69 Giulio Di Anastasio
1561 69 Giulio Di Anastasio
h1. Elimination of Duplicate points – General criteria
1562 69 Giulio Di Anastasio
1563 69 Giulio Di Anastasio
It might happen that the same physical feature (e.g. a tree, or a pole) is surveyed more than once: this can happen because there are many physical features in an area, and the survey needs more than one station. So, for example a tree is surveyed from a station, and gets a serial number on that date. When the station is then changed, it might happen that the same tree is resurveyed: another serial number is given, and possibly a different date, if the survey from the second station happened on a different day.
1564 69 Giulio Di Anastasio
It is clear that the same tree is then represented with two different points, which means that two different trees exist: but only one tree really exist in the physical reality.
1565 69 Giulio Di Anastasio
It is clear that one of the two points is redundant and needs to be removed. If this is noted by the surveyor directly in the field, then the issue is solved by the surveyor himself during processing time.
1566 69 Giulio Di Anastasio
If instead, due to various reasons, it was not noted by the surveyor in the field, it will need to be cleaned after the processing, possibly by post-processing staff.
1567 69 Giulio Di Anastasio
How to identify duplicate points?
1568 69 Giulio Di Anastasio
The following criteria can be used:
1569 69 Giulio Di Anastasio
1.	The distance between the two points is less than 30 cm (trees are surveyed if their trunk diameter is at least about 20 cm, so in 30 cm cannot exist two of them)
1570 69 Giulio Di Anastasio
2.	The orig_id (serial number) of the points are not in series
1571 69 Giulio Di Anastasio
3.	The survey date is not the same
1572 69 Giulio Di Anastasio
4.	In case of trees, the species of trees is the same
1573 69 Giulio Di Anastasio
5.	5. In case of trees, the tree type is not TDEF (because TDEF are mapped irrespective of their diameter, so they can actually have a small trunk, and two of them might exist in 30 cm), not OT (many TDEF species are surveyed as OT if not otherwise indicated by a botanist)
1574 69 Giulio Di Anastasio
6.	The context needs to be evaluated: if one tree is deleted in an area where many trees exist in a limited space, then loosing one in the map is not a big error. If instead one tree is deleted where there are very few trees, then it might be a big loss.
1575 70 Giulio Di Anastasio
1576 70 Giulio Di Anastasio
1577 70 Giulio Di Anastasio
h1. Linework for the Survey Area
1578 70 Giulio Di Anastasio
1579 70 Giulio Di Anastasio
h2. 1.	Creation of Initial Linework in QGIS using Survey points import - (Ram, System 4)
1580 70 Giulio Di Anastasio
1581 70 Giulio Di Anastasio
Initial Linework in QGIS is started by surveyor with the knowledge from the Field. For this step, points are simply imported into the QGIS from the field text file (.csv or .txt). CRS needs to be TM-AUSPOS. The box of “First record has field names” shall not be ticked. In Point Coordinates, select the correct field for x, for y and for z (usually “field_2” for x, “field_3” for y and “field_4” for z). Points can be styled using the “Categorized” style in “Symbology”, using “Field_5” as value, or using a Rule-based symbology using the category (field 5) as filter.
1582 74 Selvarani C
Linework is created by connecting points having same description and belonging to the same physical feature. *All line and polygon features are created as lines*.
1583 70 Giulio Di Anastasio
The Initial Linework for the Survey Area is also stored temporarily in 
1584 70 Giulio Di Anastasio
1585 74 Selvarani C
+D: > AVSM > Zone-Survey number (eg RZ-01) > Survey Area (eg J) > Temporary WD+
1586 70 Giulio Di Anastasio
1587 70 Giulio Di Anastasio
h2. Note: The line shapefiles / Geopackages shall be in CRS: TM AUSPOS
1588 70 Giulio Di Anastasio
1589 70 Giulio Di Anastasio
h2. 2.	Creation of final working drawing Shapefiles / Geopackages - (Selvarani, System 1)
1590 1 Philippe May
1591 1 Philippe May
Final working drawing Shapefiles / Geopackages are created from the Initial Linework of Survey Area.
1592 70 Giulio Di Anastasio
As the Surveyor draws all features as lines (both for lines and polygons features), the following actions shall be done:
1593 74 Selvarani C
1.	*If features are lines:*
1594 70 Giulio Di Anastasio
•	Export the shapefile / geopackage into the final working drawing folder (Final WD), in separate folders according to its type (e.g. BLDG, FENC, ROAD, etc).
1595 70 Giulio Di Anastasio
1596 70 Giulio Di Anastasio
h2. The CRS for the export shall be EPSG:4326 - WGS 84
1597 70 Giulio Di Anastasio
1598 74 Selvarani C
2.	*If features are polygons:*
1599 1 Philippe May
•	Lines shall be converted into polygons:
1600 74 Selvarani C
1601 1 Philippe May
to do it, first click on the layer to be converted to make it active (e.g. WD-CZ-01-F-LL84_V-BLDG-MHOL------E), then go to “Vector” Menu, click on Geometry Tools, click on Line to Polygons:
1602 87 Selvarani C
1603 84 Selvarani C
!https://redmine.auroville.org.in/attachments/download/9760/Line%20to%20Polygon%20Menu.png!
1604 70 Giulio Di Anastasio
1605 86 Selvarani C
1606 1 Philippe May
The new window for “Lines to Polygons” conversion will appear:
1607 86 Selvarani C
1608 86 Selvarani C
1609 83 Selvarani C
!https://redmine.auroville.org.in/attachments/download/9762/Lines%20to%20Polygon%20Window.png!
1610 1 Philippe May
1611 71 Giulio Di Anastasio
•	Always cross check the input layer, to make sure that the input layer is the active one
1612 71 Giulio Di Anastasio
•	Save the output in a temporary layer
1613 74 Selvarani C
•	The temporary layer will be listed in the list of layers, it shall be exported to the saving location as +D: > Survey > Zone-Survey Number > Final WD > Survey Area SHP+ (eg . D: > Survey > GB-01 > Final WD > A-Shp)
1614 72 Giulio Di Anastasio
1615 74 Selvarani C
h2. The CRS for the export shall be EPSG:4326 - WGS 84
1616 74 Selvarani C
1617 72 Giulio Di Anastasio
Once all the shapefiles / geopackages are exported in Final WD, for each of the newly exported layers the Topology Checker Tool shall be used.
1618 1 Philippe May
1619 74 Selvarani C
h2. Linework for the whole Survey Zone
1620 74 Selvarani C
1621 74 Selvarani C
h2. 1.	Merging Shapefiles / Geopackages - (Selvarani, System 1)
1622 74 Selvarani C
1623 72 Giulio Di Anastasio
A copy of the Zone Master shapefiles / geopackages are taken from System 4 and stored in Temp Folder on Desktop in System 1.
1624 1 Philippe May
Master shapefiles / geopackages are merged with the Survey Area shapefiles / geopackages:
1625 72 Giulio Di Anastasio
•	To do it, go to “Vector” Menu, click on Geoprocessing Tools, then click on Union:
1626 72 Giulio Di Anastasio
1627 85 Selvarani C
!https://redmine.auroville.org.in/attachments/download/9763/Union%20Menu.png!
1628 1 Philippe May
1629 1 Philippe May
1630 85 Selvarani C
The new window for “Union” will appear:
1631 1 Philippe May
1632 86 Selvarani C
1633 86 Selvarani C
!https://redmine.auroville.org.in/attachments/download/9764/Union%20Window.png!
1634 73 Giulio Di Anastasio
1635 74 Selvarani C
•	To make sure that the right geometry is generated by this process (“line” type, not “Multiline”, and similarly “Polygon” type, not “Multipolygon), we need to always keep the *Master shapefile* (e.g. Final-CZ-01-2021-02-05-LL84_V-BLDG-MHOL------E) *as Input layer*, and the Survey Area shapefile as Overlay Layer (e.g. WD-CZ-01-F-LL84_V-BLDG-MHOL------E).
1636 73 Giulio Di Anastasio
•	(The output can be saved to a file, as the CRS should already be EPSG4326 – WGS84.)
1637 73 Giulio Di Anastasio
1638 74 Selvarani C
h2. 2.	Storing Shapefiles / Geopackages - (Selvarani, System 1)
1639 74 Selvarani C
1640 74 Selvarani C
Save the merged shape file in the correct location in Final folder as +D: > Survey > Zone-Survey Number > Final+ (eg . D: > Survey > GB-01 > Final)
1641 73 Giulio Di Anastasio
Date in the name of Final Shapefile / Geopackage needs to be updated.
1642 73 Giulio Di Anastasio
Once the merging operation is completed, the copy of Master shapefile / geopackage is deleted from the Temp folder.
1643 73 Giulio Di Anastasio
1644 74 Selvarani C
h2. 3.	Topology check of merged shapefiles
1645 74 Selvarani C
1646 73 Giulio Di Anastasio
The topology checker is applied again on the merged shapefiles / geopackages.
1647 73 Giulio Di Anastasio
The “id_field” shall be removed from the attribute table.
1648 73 Giulio Di Anastasio
1649 74 Selvarani C
h2. 4.	Archive and replace the Master Shapefiles / Geopackages (Ram, System 4)
1650 74 Selvarani C
1651 73 Giulio Di Anastasio
Archive the previous master shapefiles / geopackages on system 4, and copy the new merged shapefiles / geopackages in its place.
1652 74 Selvarani C
*Then delete the Merged Shapefile / Geopackage folder from System 1.
1653 74 Selvarani C
*
1654 73 Giulio Di Anastasio
1655 74 Selvarani C
h2. 5.	Note about Shapefiles and Geopackages
1656 73 Giulio Di Anastasio
1657 73 Giulio Di Anastasio
All the above works are usually done using shapefile format, in QGIS latest version (3.16.3).
1658 77 Selvarani C
The Geopackage export is done in QGis versions older than 3.12 (e.g. 3.4, 3.6, 3.8, 3.10) so that the lines are not saved as “Multilines” but as “Lines”and polygons are not saved as “Multipolygons” but as “Polygons”. This is very important to be notes, as Gisaf database does not accept the Multipolygon and Multiline geometry types.