Project

General

Profile

Wiki » History » Version 72

Giulio Di Anastasio, 25/02/2021 12:15

1 1 Philippe May
h1. Wiki
2 2 Philippe May
3 48 Philippe May
h2. About the Auroville CSR Geomatics Studio
4 48 Philippe May
5 48 Philippe May
We are a team working on geographical and related information with an engineering and scientific approach. The concept of *geomatics* is explained here: https://en.wikipedia.org/wiki/Geomatics.
6 48 Philippe May
7 48 Philippe May
We develop, maintain and publish data on this web site: https://gis.auroville.org.in.
8 48 Philippe May
9 48 Philippe May
h3. Team
10 48 Philippe May
11 48 Philippe May
Currently, the team consists of:
12 48 Philippe May
13 48 Philippe May
* Bala
14 48 Philippe May
15 48 Philippe May
* Giulio
16 48 Philippe May
17 48 Philippe May
* Philippe
18 48 Philippe May
19 48 Philippe May
* Raj
20 48 Philippe May
21 48 Philippe May
* Ram
22 48 Philippe May
23 48 Philippe May
* Selvarani
24 48 Philippe May
25 48 Philippe May
h3. Collaborations
26 1 Philippe May
27 47 Philippe May
Quick report of the collaborations the Geomatics Team the areas of work: see [[collaborations]].
28 45 Philippe May
29 5 Philippe May
h2. Our workflow
30 2 Philippe May
31 5 Philippe May
See [[Workflow]].
32 2 Philippe May
33 3 Philippe May
34 3 Philippe May
h2. Working with Gisaf
35 3 Philippe May
36 6 Philippe May
h3. Survey data
37 6 Philippe May
38 12 Philippe May
Raw survey data are contained in CSV files, typically downloaded from surveying instruments.
39 12 Philippe May
40 25 Philippe May
See more information of the process for the survey data (including a flow diagram): [[Survey data]]
41 12 Philippe May
42 12 Philippe May
h3. Categories
43 12 Philippe May
44 12 Philippe May
The categories define the types of the geographical features.
45 12 Philippe May
46 12 Philippe May
See [[Categories]]
47 6 Philippe May
48 6 Philippe May
49 1 Philippe May
h3. QGis: work on shapefiles
50 1 Philippe May
51 1 Philippe May
Go to [[shapefiles]].
52 5 Philippe May
53 5 Philippe May
54 18 Philippe May
h2. Data analysis
55 18 Philippe May
56 18 Philippe May
See [[Data analysis]]
57 18 Philippe May
58 18 Philippe May
59 5 Philippe May
60 5 Philippe May
h2. Hosting
61 5 Philippe May
62 5 Philippe May
The team is located in the CSR of Auroville.
63 5 Philippe May
64 5 Philippe May
We have setup a server for hosting the software and database. See [[CSR_server]] for technical information about the setup.
65 7 Philippe May
66 7 Philippe May
67 11 Philippe May
h3. Backup and restoration of the database
68 1 Philippe May
69 1 Philippe May
See [[db-support]]
70 17 Philippe May
71 17 Philippe May
72 17 Philippe May
h2. Gear
73 17 Philippe May
74 17 Philippe May
h3. Survey equipment
75 17 Philippe May
76 17 Philippe May
See [[survey equipment]]
77 17 Philippe May
78 17 Philippe May
h3. Weather station
79 17 Philippe May
80 17 Philippe May
See [[Ambient_Weather_weather_station]]
81 11 Philippe May
82 11 Philippe May
83 7 Philippe May
h2. Plan for future
84 7 Philippe May
85 8 Philippe May
Beside living well, see [[plan]].
86 8 Philippe May
87 49 Philippe May
h2. Other
88 1 Philippe May
89 50 Philippe May
* [[GDAL (OGR) tools]]
90 49 Philippe May
91 8 Philippe May
h2. Links and references
92 8 Philippe May
93 10 Philippe May
[[links]]
94 16 Philippe May
95 16 Philippe May
96 16 Philippe May
h2. Old docs
97 16 Philippe May
98 15 Philippe May
[[Shapefiles]]
99 15 Philippe May
[[Data (measurements, auxiliary tables)]]
100 16 Philippe May
101 13 Philippe May
h2. Pavneet's docs (imported from gisaf's wiki)
102 1 Philippe May
103 15 Philippe May
104 14 Philippe May
[[Rules of Map making - What all Maps should have!]]
105 14 Philippe May
[[Survey Database]]
106 14 Philippe May
[[Field to finish]]
107 14 Philippe May
[[Survey Data Post-Processing]]
108 14 Philippe May
[[Wells Documentation]]
109 14 Philippe May
[[Civil 3D useful commands]]
110 14 Philippe May
[[Online references for Civil 3D]]
111 1 Philippe May
[[connections in QGIS- using browser panel and Add postGIS]]
112 33 Pavneet Kaur
[[Reconcilation of Raw survey data using pgAdmin]]
113 22 Pavneet Kaur
[[importing RAW data to GISAF]]
114 26 Pavneet Kaur
[[Editing Z value of features in Shapefiles in QGIS]]
115 34 Pavneet Kaur
[[Miscellaneous- Civil 3D]]
116 36 Pavneet Kaur
[[Documentation- Rain Gauge]]
117 40 Pavneet Kaur
[[Documentation- Wells Monitoring (Piezometer by Bala)]]
118 40 Pavneet Kaur
[[Documentation- Flow meter, by Bala]]
119 42 Pavneet Kaur
[[Documentation- DST- Vegetation Indexing]]
120 44 Pavneet Kaur
[[Documentation- DST- Interpolation]]
121 43 Pavneet Kaur
[[Documentation- DST- Survey- Office workflow]]
122 40 Pavneet Kaur
[[From CAD to GIS, by Giulio]]
123 41 Pavneet Kaur
[[QGIS- Miscellaneous]]
124 52 Giulio Di Anastasio
125 1 Philippe May
h2. Giulio's documentation
126 59 Giulio Di Anastasio
127 52 Giulio Di Anastasio
[[Documentation - Reconciliation of points using Gisaf]]
128 65 Giulio Di Anastasio
[[Documentation - Status and Status Changes]]
129 66 Giulio Di Anastasio
[[Documentation - Tags retained after re-import of same geometry]]
130 67 Giulio Di Anastasio
131 67 Giulio Di Anastasio
132 67 Giulio Di Anastasio
h1. Access to data
133 67 Giulio Di Anastasio
134 67 Giulio Di Anastasio
h2. Connection to server directly from CSR
135 67 Giulio Di Anastasio
136 67 Giulio Di Anastasio
To connect to the server directly without going through Aurinoco server, the correct url is
137 67 Giulio Di Anastasio
http://gis.csr.av
138 67 Giulio Di Anastasio
139 67 Giulio Di Anastasio
h2. Connection to Gisaf via QGis through WFS / OGC API
140 67 Giulio Di Anastasio
141 67 Giulio Di Anastasio
This works only on QGis from version 3.14.15 onward
142 67 Giulio Di Anastasio
143 67 Giulio Di Anastasio
In the browser, click on WFS/OGC API, then right-click to create a new connection
144 67 Giulio Di Anastasio
Give a name (e.g. OGC API Qgis Gisaf)
145 67 Giulio Di Anastasio
Give the url https://gis.auroville.org.in/ogcapi
146 67 Giulio Di Anastasio
147 67 Giulio Di Anastasio
Under the WFS Options box, on Version dropdown, the default option "Maximum" works just fine
148 67 Giulio Di Anastasio
Click on OK
149 67 Giulio Di Anastasio
The list of layers will appear in the Browser under WFS/OGC API.
150 68 Giulio Di Anastasio
151 68 Giulio Di Anastasio
152 68 Giulio Di Anastasio
h1. How to create a new projection in QGis
153 68 Giulio Di Anastasio
154 68 Giulio Di Anastasio
To create a new projection in QGis, go to menu "Settings", and click on "Custom Projections".
155 68 Giulio Di Anastasio
A pop-up window appears with a list of all projections defined in QGis projects used by the user so far.
156 68 Giulio Di Anastasio
Click on the green "+" sign on the right top part of the window to create a new projection.
157 68 Giulio Di Anastasio
In the "Name" box, type "TM CSRAUSPOS SF1" (which means TM = Transverse Mercator projection; CSRAUSPOS = theparameters for this projection are derived from the processing of DGPS raw data by AUSPOS -  Online GPS Processing Service - https://www.ga.gov.au/scientific-topics/positioning-navigation/geodesy/auspos; SF1 = Scale Factor is 1).
158 68 Giulio Di Anastasio
In the "Format" dropdown list, select "Proj String (legacy - Not Recommended)"
159 68 Giulio Di Anastasio
In the "Parameters" box, paste the following "+proj=tmerc +lat_0=12.01605433+lon_0=79.80998934 +k=1 +x_0=370455.630 +y_0=1328608.994 +ellps=WGS84+towgs84=0,0,0,0,0,0,0 +units=m +no_defs".
160 68 Giulio Di Anastasio
161 68 Giulio Di Anastasio
Finally, click on OK.
162 68 Giulio Di Anastasio
163 68 Giulio Di Anastasio
In a more explicit  way, the parameters mean the following:
164 68 Giulio Di Anastasio
Map Projection:        TransverseMercator (TM)
165 68 Giulio Di Anastasio
False Easting:          370455.6300
166 68 Giulio Di Anastasio
False Northing:        1328608.9940
167 68 Giulio Di Anastasio
Latitude of Origin:    12°00'57.79560" (DMS)   12.01605433 (DD)
168 68 Giulio Di Anastasio
Central Meridian:     79°48'35.96164"  (DMS)   79.80998934 (DD)
169 68 Giulio Di Anastasio
Scale Factor:             1.00000000
170 68 Giulio Di Anastasio
Zone Width:              6.0°
171 69 Giulio Di Anastasio
172 69 Giulio Di Anastasio
h1. Elimination of Duplicate points – General criteria
173 69 Giulio Di Anastasio
174 69 Giulio Di Anastasio
It might happen that the same physical feature (e.g. a tree, or a pole) is surveyed more than once: this can happen because there are many physical features in an area, and the survey needs more than one station. So, for example a tree is surveyed from a station, and gets a serial number on that date. When the station is then changed, it might happen that the same tree is resurveyed: another serial number is given, and possibly a different date, if the survey from the second station happened on a different day.
175 69 Giulio Di Anastasio
It is clear that the same tree is then represented with two different points, which means that two different trees exist: but only one tree really exist in the physical reality.
176 69 Giulio Di Anastasio
It is clear that one of the two points is redundant and needs to be removed. If this is noted by the surveyor directly in the field, then the issue is solved by the surveyor himself during processing time.
177 69 Giulio Di Anastasio
If instead, due to various reasons, it was not noted by the surveyor in the field, it will need to be cleaned after the processing, possibly by post-processing staff.
178 69 Giulio Di Anastasio
How to identify duplicate points?
179 69 Giulio Di Anastasio
The following criteria can be used:
180 69 Giulio Di Anastasio
1.	The distance between the two points is less than 30 cm (trees are surveyed if their trunk diameter is at least about 20 cm, so in 30 cm cannot exist two of them)
181 69 Giulio Di Anastasio
2.	The orig_id (serial number) of the points are not in series
182 69 Giulio Di Anastasio
3.	The survey date is not the same
183 69 Giulio Di Anastasio
4.	In case of trees, the species of trees is the same
184 69 Giulio Di Anastasio
5.	5. In case of trees, the tree type is not TDEF (because TDEF are mapped irrespective of their diameter, so they can actually have a small trunk, and two of them might exist in 30 cm), not OT (many TDEF species are surveyed as OT if not otherwise indicated by a botanist)
185 69 Giulio Di Anastasio
6.	The context needs to be evaluated: if one tree is deleted in an area where many trees exist in a limited space, then loosing one in the map is not a big error. If instead one tree is deleted where there are very few trees, then it might be a big loss.
186 70 Giulio Di Anastasio
187 70 Giulio Di Anastasio
188 70 Giulio Di Anastasio
h1. Linework for the Survey Area
189 70 Giulio Di Anastasio
190 70 Giulio Di Anastasio
h2. 1.	Creation of Initial Linework in QGIS using Survey points import - (Ram, System 4)
191 70 Giulio Di Anastasio
192 70 Giulio Di Anastasio
Initial Linework in QGIS is started by surveyor with the knowledge from the Field. For this step, points are simply imported into the QGIS from the field text file (.csv or .txt). CRS needs to be TM-AUSPOS. The box of “First record has field names” shall not be ticked. In Point Coordinates, select the correct field for x, for y and for z (usually “field_2” for x, “field_3” for y and “field_4” for z). Points can be styled using the “Categorized” style in “Symbology”, using “Field_5” as value, or using a Rule-based symbology using the category (field 5) as filter.
193 70 Giulio Di Anastasio
Linework is created by connecting points having same description and belonging to the same physical feature. All line and polygon features are created as lines.
194 70 Giulio Di Anastasio
The Initial Linework for the Survey Area is also stored temporarily in 
195 70 Giulio Di Anastasio
196 70 Giulio Di Anastasio
D: > AVSM > Zone-Survey number (eg RZ-01) > Survey Area (eg J) > Temporary_ Zone-Survey number-Survey Area-YYYY-MM-DD (Temporary_GB-01-A-2019-06-03). 
197 70 Giulio Di Anastasio
198 70 Giulio Di Anastasio
The Initial Linework is made of either lines or polylines (closed polylines for polygon geometry) to be correctly exported as shapefiles or geopackages.
199 70 Giulio Di Anastasio
200 70 Giulio Di Anastasio
The line shapefiles are stored in WD folder. 
201 70 Giulio Di Anastasio
D: > Survey > Zone-Survey Number > WD > Survey Area SHP
202 70 Giulio Di Anastasio
eg . D: > Survey > GB-01 > WD > A-Shp
203 70 Giulio Di Anastasio
204 70 Giulio Di Anastasio
h2. Note: The line shapefiles / Geopackages shall be in CRS: TM AUSPOS
205 70 Giulio Di Anastasio
206 70 Giulio Di Anastasio
h2. 2.	Creation of final working drawing Shapefiles / Geopackages - (Selvarani, System 1)
207 70 Giulio Di Anastasio
208 70 Giulio Di Anastasio
Final working drawing Shapefiles / Geopackages are created from the Initial Linework of Survey Area.
209 70 Giulio Di Anastasio
As the Surveyor draws all features as lines (both for lines and polygons features), the following actions shall be done:
210 70 Giulio Di Anastasio
1.	If features are lines:
211 70 Giulio Di Anastasio
•	Export the shapefile / geopackage into the final working drawing folder (Final WD), in separate folders according to its type (e.g. BLDG, FENC, ROAD, etc).
212 70 Giulio Di Anastasio
213 70 Giulio Di Anastasio
h2. The CRS for the export shall be EPSG:4326 - WGS 84
214 70 Giulio Di Anastasio
215 70 Giulio Di Anastasio
2.	If features are polygons:
216 70 Giulio Di Anastasio
•	Lines shall be converted into polygons:
217 70 Giulio Di Anastasio
to do it, first click on the layer to be converted to make it active (e.g. WD-CZ-01-F-LL84_V-BLDG-MHOL------E), then go to “Vector” Menu, click on Geometry Tools, click on Line to Polygons:
218 70 Giulio Di Anastasio
!Line to Polygon Menu.png!
219 71 Giulio Di Anastasio
220 71 Giulio Di Anastasio
The new window for “Lines to Polygons” conversion will appear:
221 71 Giulio Di Anastasio
!Lines to Polygon Window.png!
222 72 Giulio Di Anastasio
223 72 Giulio Di Anastasio
•	Always cross check the input layer, to make sure that the input layer is the active one
224 72 Giulio Di Anastasio
•	Save the output in a temporary layer
225 72 Giulio Di Anastasio
•	The temporary layer will be listed in the list of layers, it shall be exported to the saving location as D: > Survey > Zone-Survey Number > Final WD > Survey Area SHP (eg . D: > Survey > GB-01 > Final WD > A-Shp)
226 72 Giulio Di Anastasio
•	The CRS for the export shall be EPSG:4326 - WGS 84
227 72 Giulio Di Anastasio
228 72 Giulio Di Anastasio
Once all the shapefiles / geopackages are exported in Final WD, for each of the newly exported layers the Topology Checker Tool shall be used.
229 72 Giulio Di Anastasio
230 72 Giulio Di Anastasio
Linework for the whole Survey Zone
231 72 Giulio Di Anastasio
1.	Merging Shapefiles / Geopackages - (Selvarani, System 1)
232 72 Giulio Di Anastasio
A copy of the Zone Master shapefiles / geopackages are taken from System 4 and stored in Temp Folder on Desktop in System 1.
233 72 Giulio Di Anastasio
Master shapefiles / geopackages are merged with the Survey Area shapefiles / geopackages:
234 72 Giulio Di Anastasio
•	To do it, go to “Vector” Menu, click on Geoprocessing Tools, then click on Union:
235 72 Giulio Di Anastasio
236 72 Giulio Di Anastasio
!Union Menu.png!