Wiki » History » Version 272
Version 271 (Giulio Di Anastasio, 06/05/2021 10:58) → Version 272/317 (Giulio Di Anastasio, 06/05/2021 11:07)
h1. %{color:BLUE} Wiki%
h1. [[CSR Geomatics Team]]
h1. [[Collaborations]]
h1. [[Data Analysis]]
h1. [[CSR Geomatics Server Setup]]
h1. [[Database Backup and Restoration]]
h1. [[CAD to GIS Workflow]] %{color:RED} - TO BE UPDATED%
h1. [[GIS Conventions]]
h1. [[Working with Gisaf]]
h1. [[Gisaf Workflow]]
h1. [[GIS to CAD Workflow]] %{color:RED} - TO BE UPDATED%
h1. [[Weather Station]]
h1. [[GDAL - Useful Tips]]
h1. [[Gisaf - Auxiliary Data (non-geographical data)]]
h1. [[QGis - Useful Tips]]
h1. [[AutoCAD Civil 3D - Useful Tips]]
h1. [[Wells documentation]]
h1. [[Gisaf Data Modifications]]
h1. Plan
Some interesting projects that might be integrated:
* https://github.com/Oslandia/albion : Build 3D geological model from wells information
h1. Links
h2. Water management
h3. Modflow
* https://water.usgs.gov/ogw/modflow/
The reference software for underground water modelling and simulation.
In conjunction with flopy (https://water.usgs.gov/ogw/flopy/) and Jupyter (https://jupyter.org/), it provides a _relatively easy_ to use interface.
h3. Freewat
* http://www.freewat.eu/project
This project is partly based on modflow, and integrates with QGis.
h2. QGIS
* https://www.qgis.org/en/site/
* Tools for Geology
Construction of geological cross sections in QGIS - http://www.geokincern.com/?p=1452
h2. Autocad
* Overview of Converting Geospatial Data to Drawing Objects:
http://docs.autodesk.com/CIV3D/2013/ENU/index.html?url=filesMAPC3D/GUID-C38FD485-3CC2-4B52-8264-0D8C0F45422B.htm,topicNumber=MAPC3Dd30e41809
* CAD-DB connection:
https://knowledge.autodesk.com/support/autocad-civil-3d/learn-explore/caas/video/youtube/watch-v-AQoB--nyUJA.html
h1. Orfeo
* https://www.orfeo-toolbox.org/
Remote sensing
h2. Reconciliation of points
The points are noted with their codes from and codes to reconcile into. ADD!! - *%{color:RED} to be written%*
h1. Survey Data Post-Processing
+Softwares used: Civil 3D, QGIS+
AutoCAD Civil 3D - C3D,
C3D is being used for post processing of survey data. C3D offers a BIM solution for Land Surveyors. Basic knowledge about C3D can be obtained by following tutorials online though a pre-acquired knowledge of Autodesk AutoCAD is a plus point and sufficient enough to start working with C3D.
QGIS is an Open source GIS.
Getting Started:
+Standards and Workflow for C3D+
The layers (Nomenclature of layers is coming from standards- U.S. National CAD Standard Version-AIA) are pre-assigned a geometry type (Point, Line and Polygon) in their layers description (Use layer manager to check). This is done keeping in mind inter-operability with GIS (In shapefiles format), geometry type Shape files are of three kinds- points, lines and polygons.
+Geometry types-+
Based on the description of Layers, if the layer Geometry is specified as 'Point', the raw survey data can be auto imported to webGIS - *GISAF*, no post processing is needed for 'Points'. For lines and polygons, We use Polylines (for 2D lines/Polygons) and 3Dpolylines (for 3D lines/polygons). In case of curved lines in 2D, Polylines command (PLINE/PL) is enough but its not possible to create curved lines in 3D using the 3Dpolylines, in such cases feature lines come into the picture. (*Important:* 2D polylines, splines, ellipses and circles are not exportable to shapefile format, they have to be always converted into Polylines).
*We are at the moment generating shapefiles (lines and polygons) in 2D (z=0) but keeping the format of files as 3D.
+Using Feature lines for creating Curved geometries in 3D-+
There are a series of steps involved when creating curved lines using feature lines. You can follow two methods depending on the situation :-
Method 1(For curved objects in 3D using elevation of points):- Create the object using 3DPOLY. Now use the create feature lines from objects and select the objects to be converted to feature lines. You will not notice any difference in the geometry on the screen as such but in properties you will see the object type description of selected object as feature lines.
Method 2(For curved objects in 3D by draping the lines/curves on a surface):- Create the object using PLINE. You will notice all the lines are straight. Now use the create feature lines from objects and select the objects to be converted to feature lines. You will not notice any difference in the geometry on the screen as such but in properties you will see the object type description of selected object as feature lines.
Now modify tab is used to convert the straight lines to curved lines. There are many ways to do it and different methods can be adopted in different situations. The easiest and the most commonly used method is by using the smooth command in the modify tab. After using the smooth command, a curve would be visible. This is a curve in 3D. To view it, object viewer option can be used by selecting the object and right clicking.
More complex methods have to be studied and each case has to be taken into consideration separatly when using - FITCURVEFEATURE In some cases to obtain a desired curve(Meaning more fragments) more PI's (Point of insertion) need to be added into the feature line. PI are the points where the feature lines gets fragmented when exploded.This step often becomes difficult to manage due to complexities involved in mathematical functions behind creation of feature lines.But it should be explored by all means.
Because the feature lines are not exportable to shapefiles, the feature line has to converted back into a 3D polyline. This can be done simply by explolding the object. Once exploded into 3D polylines, you will see that the curve gets fragmented into smaller segments of straight lines. You can repeat the process of creating the 3Dpolyline object into feature line and then smoothening it till you get a desired geometry resembling the curve.
Situations of sharing of points with different layers and proceeding with linework in such circumstances-
There is always a best way to take the survey points taken in the field by the surveyor. For instance, in case of a road and a curb adjacent to it, the surveyor takes the points only once. It is only during the post processing that the lines are generated (You can create the lines in field while taking the survey points as well, but that is not the most feasible method majorly due to time constraints) and in this case, two line on the same position shall be generated, one under layer of roads and one under layer of curbs.
The line work becomes complex in case of generating 3d lines and 2d lines on the same place. This can arise due to having some layers as 2D (for example building outlines) and some as 3D (paved surfaces- <taking into consideration the future use of the layers. In this case, for the purpose of water management, it becomes essential to have survey of such surfaces as 3D. Survey of building outlines doesn't need to be 3D because the elevation points are not recorded while surveying them due to feasibility).
So, in case of generating 3D linework from survey points with no elevation, Feature lines and SURFACES come into picture. The surfaces are generated by an interpolation method in C3D by giving a set of Point Groups. We have selected the triangulation method as the appropriate method due to availability of a dense set of points.
While creating the feature lines from objects (as explained above, check the box "assign elevation" and a dialog box appears to select the surface you want to select). Surfaces will have to be generated prior to using this. (add wiki for generating surface). Surface should be generated including the points of the layer in the point group of it.
*+Purpose of Survey+*
# The survey is conducted for generating a Base Map which is representative of Topography, Infrastructure, Drainage, Buildings etc, with a focus on Water Management in Auroville. Furthermore this survey can be used as base map to build upon more detailed surveys for the various purposes such as Town planning, Land Surveys, Transportation Planning etc.
+Use of Layers+
# *V-BLDG-SHED* (Shed) layer is used when the structure is not entirely enclosed, if At least one side is open (No walls/partition etc). One thumb rule is to answer the question if the building can be locked safely or not. If not, then it is a shed.
# *V-BLDG-HUT* (Hut) is used when the building (For human activity/living space) is single storey with roof made up of perishable material such as keet, straw etc.
# Use of *V-BLDG-ROOF* (Roof) is for buildings with permanent roofs extending out about more than 1 meter from the outline at ground level. It can also be used for complex roof structures for example Matrimandir Petals.
# Use of *V-BLDG-OTLN* (Building outline) is for depicting the outer line of walls of a structure on ground level. This is used when the roof of the building is more or less of the same profile.
# *V-BLDG-RTWL* (Retaining Wall) is for walls with the function of retaining either earth/water and the side of the wall should be visible. In case of very thin walls and no side visible, it comes as *V-BLDG-RWLL* (Retaining wall line). This is depicted as a single line in drawing.
# Use of *V-ROAD-CYCP* is for designated cycle path as a single line.
# Use of *V-ROAD-FPAT* is for pedestrian paths as a single line. It is used when there is a path of very small width. If the need is to represent a path/road which is unpaved in nature, *V-ROAD-UPVD* is used, which is polygon type. To represent a path/road of paved nature *V-ROAD-PAVD* is used, which is also a polygon type.
# *V-WMNG-PIPP* for pipe points, *V-WMNG-PIPL*- for pipe lines.
# *V-BLDG-BMRK* is for unidentified bench marks and *V-BLDG-MHOL* is for unidentified manholes.
# *V-BLDG-PLTF* is for impervious surfaces. for example sand and gravel will not come under this layer. cemented paths etc can come.
# *V-WMNG-POOL* is for water bodies made for leisure activities- like swimming pool etc.
# *V-WMNG-SUMP* is collectively for artificial water retention structures.
# *V-WATR-POND* is for top of natural water retention.
# *V-WATR-PONB* is for bottom of natural water retention.
# *V-WMNG-DRBA* is for bottom of artificial drains.
# *V-WMNG-DRTA* is for top of artificial drains.
# *V-WATR-DRBN* is for bottom of Natural drains.
# *V-WATR-DRTN* is for top of Natural drains.
# *V-WMNG-DRNC* is for drain covers.
# *V-WATR-DRAS* is for indicating slope of drainage, these are arrows.
h1. *Reconcilation of Raw survey data using pgAdmin*
+For changing the layer codes, to be done using Pgadmin.+
h3. Enter Schema> AVSM RAW Survey > Tables > select the table> Right click - View edit data > All rows. Apply filter to the original id and note the Database id's to be reconciled.
h3. GISAf Admin> Other > Reconciliation. Create > Add the database (point) id and give the new target corresponding to the database id to be changed. (Use the Layer name in target not Raw layer name).
h3. Other > Project > select the project > with selected> reconcile RAW survey points.
h1. Importing point data (TS and RTK) to GISAF
# *Gisaf Admin > Basket > Survey data > "Project" > "surveyor" > TS/ RTK > upload > import.*
# *Auto import of Raw points data* (changes from Raw to Shapefiles, the point files)
In GISAF Admin > Other> Project > select the project for which you want to import the data > with selected > Auto import to GIS...
h1. Editing Z value of features in Shapefiles in QGIS
h2. using vertex editor tool - https://www.youtube.com/watch?v=8V8i1AtcA74&t=256s
h1. Miscellaneous- Civil 3D
*Autodesk civil 3D Geotechnical module- for borehole data :- To analyse borehole data, To make profiles and calculate volumes*
https://www.youtube.com/watch?v=0Be9kShBou0
Reference videos
Exploring:
https://www.youtube.com/watch?v=gr-ISPzLcU0
Modelling:
https://www.youtube.com/watch?v=Pz0-HOoiBrs
solids
https://www.youtube.com/watch?v=mQ0Yeh6tZA8
https://www.youtube.com/watch?v=4eMsUiYBhuE
*CIVIL 3D Survey*
Getting started- Always open a new drawing with a template.
*Description key sets*- (till 18:30) https://www.youtube.com/watch?v=mmwkkRyBkS0
As when the points come into the drawing, they are going to be filtered with description key sets and the description key sets will assign properties to the points.
Tool space > settings > points > description key sets
“Description key sets name”> edit key- to view the points in a list:
Automatic linework (18:30- till end)
Survey > Linework code sets- for automatic linework
*Using Master view*
Uses-
copy styles from one drawing to other
copy drawing data from one drawing to other (data referencing)
*Civil 3d surface model*- from points- break lines
https://www.youtube.com/watch?v=wC40rdmDEMo
moving for example- point groups under point groups-
*Civil 3D Planning and Analysis*
Workspace: Planning and Analysis
1. Working with Object data (GIS Attributes)
Map Setup> Define object data> New table> Define new object data table start defining fields
h1. Documentation- Rain Gauge
h2. Manual Rain Gauge
*Why it is important to comply to standards?*
1. Consistency for comparing rainfall data in different places within Auroville since rainfall varies in different parts of Auroville.
2. Making it possible to use for any kind of scientific analysis by contribution to the primary data in sustainable water management for Auroville.
*Proposed standards and ethics:*
1. The time of taking the reading - 08:30 am.
The rain is recorded over a period of 24 hours- 8:30 of previous day till 8:30 of the present day, and the date is put as the present day. For example, if the rain is measured at 8:30 am on 10 dec 2017, then it is logged in as on 10 dec 2017.
2. Manual rain gauge typically used- green cylindrical and a collection jar.
3. Measuring jar: 10 mm corresponding jar
Area of the rim of manual rain gauge= 200 square cm
Diameter of rim= approximately 16 cm
*NOTE:* - If another kind of rain gauge is used, it should be used with the corresponding measuring jar as type of jar and calibration on jar depends on the area of rim. Any cases of different rain gauge other than above should be reported before use for recording purposes.
4. The minimum record-able unit is 0.2 mm (Least measure on the 10 mm cylinder). T is marked for below 1 mm.
5. The reading should be preferably submitted in an excel sheet (the data sheet will be provided). The frequency of sending data can be daily or weekly.
6. The rain gauge should be kept in a safe and open to sky area clear from tree cover or objects in surrounding so that direct rain falls into it and not from trees/ objects etc. Preferably roof top or a higher place on ground.
7. A check on the rain gauge should be kept when there is long period of gap in rainy days to make sure it was emptied out and there is no blockage.
8. In case of doubt in reading or incorrect measure, "incorrect measure" should be marked
*Ethics*
1. Readings should not be missed. Time to time check on rain gauge is required during long periods of no rain (especially when the rain season is approaching).
This is important because firstly, if it rained in the night and it wasn’t recorded, reading for the day is lost; Secondly, if the jar was not emptied, it gives incorrect subsequent reading.
2. If the person is not going to be available temporarily where the rain gauge is kept, the responsibility should be passed on to someone they can rely on after explaining the standards.
*Important-*
Data won’t be published in case of non-compliance to standards. This is to maintain the sanctity of proper scientific data collection and to keep it reliable for sharing by publishing. Discussions regarding standards are welcomed.
CSR Geomatics Team is placed on first floor in CSR, Auroshilpam.
*AV rain data publication*: (http://gis.auroville.org.in/measures/raingauge_av).
Adding Rain gauge to our web portal (GISAF), following information is needed:
1. GPS co-ordinates of location of placing the rain gauge (can be obtained using mobile phones easily)
2. Name, place of residence & contact number- Mobile & Landline
h2. Automatic Rain Gauge
There is a possibility of publishing rain data coming from Automatic rain gauges like (id 15, auro orchard) and Weather stations.
For Automatic rain gauges, we can upload the files coming from the rain gauge directly into GISAF.
_Thank you for contribution towards a sustainable management of Water in Auroville through Data collection._
h1. Documentation- Wells Monitoring (Manual and Automatic / Piezometer by Bala)
h2. Manual- Using tape with a sensor
Timings: The person assigned the job of monitoring collects the readings in three slots.
1. Between 6 am to 7 am till about 9 am to 10 am.
2. Between 11 am and 1 pm
3. Between 2 pm and 5 pm
Things required:
• Notebook
• Pen
• Measuring tape
• Vehicle for movement
*Ethics*-
• The monitoring should happen in coordination with the community members/ care taker (assigned by the person in charge from within the community). There should be a clear communication from monitor side as to which days the monitoring of a well has to happen and at around what time so that it can be made sure that the pump is not turned on before monitoring. In case a pump was turned on in a well, the monitor should have a gap of about 5 hours on the same day before going again for monitoring.
• The monitor should have contact numbers of the person in charge/ care taker for any communication.
• If the monitoring is stopped for any reason at any point, the monitor should communicate the same to the person concerned.
• The monitor shall take responsibility to inform any kind of changes in a well in terms of its functioning etc.
• The monitor should report to the CSR Geomatics Team who has responsibility to publish data.
• The monitor published the data to the website and works with the geomatics team.
*Definitions*:
A master file is maintained with the records of the wells from the field. It is updated when a new well is located. The following set of information are filled out in the Masterfile.
*Well type*
1. Open
2. Borewell
3. Dug cum borewell
*Well status*
1. In use
2. Not in use
3. Closed
*Pump Status*
1. Functioning
2. Not functioning
*Pump Automation*
1. Manual
2. Automatic
* Non-accessibility factors*
1. Dogs
2. Heavy slab
3. Narrow casing
4. enclosed/locked
5. permission
*Data Matching Accuracy (reconnecting with harvest wells data)*
1. High
2. Medium
3. Low
*Well coordinates*
For a new well, coordinates of the well position are taken on a mobile GPS.
Person In charge
For communication purpose, contact the person in charge as recorded in directory.
h2. Automatic- Piezometer (by bala, to be edited)
The calibrations and setting are already done by Azha
The piezometer is taken to the site of unused well. First the depth and Water level is checked manually using water meter
Then the sensor of the piezometer is inserted into the well 1m above the bottom of the well, so the sensor does not get affected from mud or water inside
The sensor sends the data to an electronic board which translates the signal into the proper output, and then it sends the signal to the transmitting device above, on the ground surface. The transmitter then sends it to the receiver at CSR, the signal is then routed to Talam office through Interneet, enters the software and the reading is processed.
The piezometer sometimes does not have proper signal and so it would not be able to send it
h1. Documentation- Flow meter by Bala
Flow meter – for checking the flow of water in pipe
CSR bought ultrasonic flow meter from Chennai and the company (company name? )trained Bala and Vijai(CSR) on how to set it up.
we tested in the west water system pump in csr. then we test in many more place .
ami, aurodam and buddha garden borewell pumps . we got request from the water service
Cross check they flow meters .
so before that we want to know how our meter works.
so we tested in our tank . the pump pumping from our sump tank so the flow will be stranded. when you pump in the bore well the flow goes up and down.
we tested two times in 15 minutes.and one time 30 minutes,so we got variation between this three.
we find 1,5 % error but the flow meret they said 1 % error only.
we cross checked the flow meter of the water service we find some error in they meter also.
we tried they bore well flowmeter and they sump tank flow meter also.and we find some error also in they meters.
then we find some error in (AVWS) meter also.
It is used to check the flow of water in pipe
tools used
First the outer diameter of the pipe has to be entered in the device. It is measured using Vernier caliper
Then the thickness of the pipe is set which is also measured using vernier
Also the device asks for the material of the pipe. If the pipe material is know it can be set and if it not know then there is an option which is other (mostly PVC, HDPE, and iron)
After entering these details the device gives the spacing for the sensors. There are 2 sensors up and down which has to be fixed accordingly
When the sensors are fixed the motor is turned on and the sensors send reading to the display device
This is noted down once every minute and taken for 15 to 30 minutes. Then the average of this is determined. This is done because of the variation in the flow. This gives the flow rate in 1 hour.
This was done in different places to check to flow rate
h1. Documentation- DST- Vegetation Indexing
h2. Steps for Dzetsaka Classification tool for Vegetation indexing in QGIS
1. Install the plugin Dzetsaka classfication tool.
2. Open the Raster from the Survey.
3. Create a polygon shapefile for index sampling. Mark polygons and give the ID's (1,2,3 for Tree, grass, bare land etc) Cover the variations in samples as much as possible.
More the samples, better the indexing.
4. Apply Dzetsaka Classification tool, Select the base raster and the sample- index polygon shapefile created in step 3.
5. The result is a Raster with DN numbers specified in the Shapefile in step 3.
6. Apply the Sieve raster command (Raster> Analysis> Sieve)- Try different threshold numbers and view the results till the noise is removed from the Raster.
7. Polygonise the Raster to Vector (From processing)
8. Run the v.generalise tool on the shapefile. This tool removes the pixelated boundaries of the polygons in the Vector.
h1. Documentation- DST- Interpolation (Processing toolbox)
The following 4 tools have been mostly explored and the results were compared. The ones used for quick analysis are 1. Cubic Spline and 2. V.surf.spline . The rest of the tools are for further exploration and used depending on the need of the project. In some tools, the elevation values of points should be stored in the attribute table (Using field calculator and giving command - *Z($Geometry)* )
# Interpolate (Cubic spline) - SAGA
# V.surf.bspline - GRASS. Parameters to set - cell size. Set this parameter above 0.001 ( 0.00001, 0.000001 etc) and check the results.
# V.surf.rst - GRASS
# Krigging - SAGA
[[Documentation- DST- Survey- Office workflow]] - *%{color:RED} to be written%*
h1. From CAD to GIS by Giulio
*FEATURES CREATION*
1. Assign a CRS to the drawing (TM-AUSPOS) (MAPCSLIBRARY command)
2. Create features in CAD (Points, lines, polygons)
3. Export shapefile (a) from CAD (Output > DWG to SDF) (Convert to LL84 – 3D)
*FEATURES IMPORT INTO DB FIRST TIME*
4. Create zip file of the shapefile
5. Upload into the GISAF Shapefiles Basket
6. Import the shapefile into DB
7. Save the shapefile on Local Machine
*FEATURES IMPORT INTO DB EVERYTIME*
8. Combine the new features to corresponding last shape files (Insert the process here).
9. Follow step 4-8 again
*FEATURES EDITING IN QGIS*
10. Open the table in QGis
11. Save as a shapefile (b) in TM AUSPOS CRS
12. In CAD, open a new drawing and assign AUSPOS CRS
13. Import the shapefile (b) (MapImport) with Object Data (Data tab > Create Object Data > OK), tick “Import polygons as closed polylines”, then press OK
14. Edit features
15. Change workspace into “Planning and analysis”
16. Export shapefile (a) from CAD (Output > DWG to SDF) with ONLY the id selected (Data Tab > Select Attributes > Object Data > Filename > id) (Convert to LL84 – 3D)
*FEATURES IMPORT INTO DB*
17. Create zip file of the shapefile
18. Upload into the GISAF Shapefiles Basket
19. Import the shapefile into DB
20. Delete the shapefile from Local Machine
h1. QGIS- Miscellaneous
*QGIS Introduction:*
https://www.birdseyeviewgis.com/blog/2018/2/22/my-favorite-features-of-qgis-30to-date
3D visualization of raster DEM- https://www.youtube.com/watch?v=2KrCsbP0kUs
*Spatial query:*
Spatial Query is selection of features that satisfies a certain condition which relates to other features in a space.
Using plugin- Spatial query
http://www.geodose.com/2018/03/spatial-query-in-qgis-3.html
Labelling:
Labelling with more than one field names and in different lines
|| '\n' ||
*Hierarchy of extensions*
File levels and their uses.
https://nathanw.net/2014/03/22/all-the-q-files/
+The Project file (.qgs)+
It contains: Layer source pointer + Style information + Composers + a whole heap of other stuff
+The Layer Definition file (.qlr)+
It contains: Layer source pointer + Style information
+The QML file (.qml)+
It contains: Style information
h2. Giulio's documentation
h1. Documentation - Reconciliation of points using Gisaf
Reconciliation of points is a procedure used when a point is stored in a wrong table, because its category was either wrongly recorded in the field by the surveyor, or it has ben reviewed later by the surveyor or the data validator and found to be wrong.
h2. Definition: Raw points are all points coming from the field survey. Raw points can be points referring to a Point feature (e.g. trees, or elevation points, or floor level), or points measured in the field to draw a line (e.g. the vertices of a fence) or a polygon (e.g. the corners of a building outline).
In the overall workflow, *+reconciliation affects only raw points+*. It takes place after the field textfile is uploaded into the basket and its points imported (raw points stored in the raw survey tables). Here you can visualize the Workflow diagram: [[Survey_data]].
If a raw point refers to a point feature, reconciling it means moving it to another category/table meant for point features, not for lines/polygons. So, *+only raw points of point features can be reconciled+*. Raw points pertaining to line features and/or polygon features cannot be reconciled, so these raw points will remain in their original wrong table.
PLEASE NOTE:
The attributes of a line/polygon, which are derived from their defining raw points, will not be modified by any reconciliation, because *+only raw points of point features can be reconciled+*.
h2. How to perform Reconciliation
To perform reconciliation of points: Login to Gisaf -> click on the G icon on the upper-left corner of the website page -> Manage -> Reconciliation by orig.ID
On the right end side of the screen, clicking on the field "Destination" a list of all categories will appear: these categories refer not to the Raw survey tables, but to the V_ tables of the database (points, lines, polygons).
Under it, in the field "Original ID", the original point number of the point to be reconciled is to be entered.
Clicking on the "Search points" button, the result shows the database unique id of the point, its survey category, its survey date, its geometry type (point, line, polygon), and the Project the point belongs to. In case of multiple points with the same original id (in case of different Projects, the field number of points might be not unique if the numbering of points in the field has restarted) all points having that original id are displayed: thanks to their date or Project or type, it is easy to identify the correct point to be reconciled.
Once the point to be reconciled is identified, clicking on the button "Reconcile" will run the reconciliation, and a message will appear stating that it has been done successfully. An error message can appear if a reconciliation of a raw point of a line/polygon feature has been attempted: this type of points cannot in fact be reconciled.
Once a raw point has been reconciled, *+it cannot be reconciled again+*. In case a raw point has been wrongly reconciled, it cannot be reconciled again through the above procedure, but it has to be reconciled manually through QGis or pgadmin software.
h1. Documentation - Status and Status Changes
Status have been created to keep track og changes in surveyed features.
It is an additional (though provided for in AIA standards), single digit value, at the end of the Gisaf Category/Cad layer name.
Each Status need to have a corresponding CAD layer/Gisaf Category, with a short code associatedto it, so that field entries can be done easily.
Status have been defined as follows:
N - New Work
E - Existing o remain
D - Existing to demolish, Demolished or Changed
F- Future work, Proposed feature
T - Temporary work
M - Item to be moved
X - Not in contract
By default, Status is defined as E (Existing). Status can anyhow be changed later using gAdmin, or QGis (through the PostGis connection).
It needs to be done manually, one feature (point, line, polygon) at a time.
In the future Status changes might be incorporated in the Admin panel.
h1. Documentation - Tags retained after re-import of same geometry
Documentation - Tags retained after re-import of same geometry
The linework for infrastructure survey carried out by Eric Chacra in May 2020 was originally imported with a problem of ambiguity in the "Accuracy" table and in the "Accuracy" table.
The result was that lines did not inherit the attributes survey date, accuracy, equipment, surveyor.
Nevertheless lines were displayed on the Gisaf map, without these attributes, and tags were given to some of these lines.
The values for the two tables ("Accuracy", "Accuracy") have been corrected, ambiguity resolved.
The lines in the layers V-ELEC-UGND------E and V-COMM-CABL------E have been then reimported, the attributes have been properly assigned, and the tags have been retained.
4 August 2020
h1. Access to data
h2. Connection to server directly from CSR
To connect to the server directly without going through Aurinoco server, the correct url is
http://gis.csr.av
h2. Connection to Gisaf via QGis through WFS / OGC API
This works only on QGis from version 3.14.15 onward
In the browser, click on WFS/OGC API, then right-click to create a new connection
Give a name (e.g. OGC API Qgis Gisaf)
Give the url https://gis.auroville.org.in/ogcapi
Under the WFS Options box, on Version dropdown, the default option "Maximum" works just fine
Click on OK
The list of layers will appear in the Browser under WFS/OGC API.
h1. How to create a new projection in QGis
To create a new projection in QGis, go to menu "Settings", and click on "Custom Projections".
A pop-up window appears with a list of all projections defined in QGis projects used by the user so far.
Click on the green "+" sign on the right top part of the window to create a new projection.
In the "Name" box, type "TM CSRAUSPOS SF1" (which means TM = Transverse Mercator projection; CSRAUSPOS = theparameters for this projection are derived from the processing of DGPS raw data by AUSPOS - Online GPS Processing Service - https://www.ga.gov.au/scientific-topics/positioning-navigation/geodesy/auspos; SF1 = Scale Factor is 1).
In the "Format" dropdown list, select "Proj String (legacy - Not Recommended)"
In the "Parameters" box, paste the following "+proj=tmerc +lat_0=12.01605433+lon_0=79.80998934 +k=1 +x_0=370455.630 +y_0=1328608.994 +ellps=WGS84+towgs84=0,0,0,0,0,0,0 +units=m +no_defs".
Finally, click on OK.
In a more explicit way, the parameters mean the following:
Map Projection: TransverseMercator (TM)
False Easting: 370455.6300
False Northing: 1328608.9940
Latitude of Origin: 12°00'57.79560" (DMS) 12.01605433 (DD)
Central Meridian: 79°48'35.96164" (DMS) 79.80998934 (DD)
Scale Factor: 1.00000000
Zone Width: 6.0°
h1. Elimination of Duplicate points – General criteria
It might happen that the same physical feature (e.g. a tree, or a pole) is surveyed more than once: this can happen because there are many physical features in an area, and the survey needs more than one station. So, for example a tree is surveyed from a station, and gets a serial number on that date. When the station is then changed, it might happen that the same tree is resurveyed: another serial number is given, and possibly a different date, if the survey from the second station happened on a different day.
It is clear that the same tree is then represented with two different points, which means that two different trees exist: but only one tree really exist in the physical reality.
It is clear that one of the two points is redundant and needs to be removed. If this is noted by the surveyor directly in the field, then the issue is solved by the surveyor himself during processing time.
If instead, due to various reasons, it was not noted by the surveyor in the field, it will need to be cleaned after the processing, possibly by post-processing staff.
How to identify duplicate points?
The following criteria can be used:
1. The distance between the two points is less than 30 cm (trees are surveyed if their trunk diameter is at least about 20 cm, so in 30 cm cannot exist two of them)
2. The orig_id (serial number) of the points are not in series
3. The survey date is not the same
4. In case of trees, the species of trees is the same
5. 5. In case of trees, the tree type is not TDEF (because TDEF are mapped irrespective of their diameter, so they can actually have a small trunk, and two of them might exist in 30 cm), not OT (many TDEF species are surveyed as OT if not otherwise indicated by a botanist)
6. The context needs to be evaluated: if one tree is deleted in an area where many trees exist in a limited space, then loosing one in the map is not a big error. If instead one tree is deleted where there are very few trees, then it might be a big loss.
h1. Linework for the Survey Area
h2. 1. Creation of Initial Linework in QGIS using Survey points import - (Ram, System 4)
Initial Linework in QGIS is started by surveyor with the knowledge from the Field. For this step, points are simply imported into the QGIS from the field text file (.csv or .txt). CRS needs to be TM-AUSPOS. The box of “First record has field names” shall not be ticked. In Point Coordinates, select the correct field for x, for y and for z (usually “field_2” for x, “field_3” for y and “field_4” for z). Points can be styled using the “Categorized” style in “Symbology”, using “Field_5” as value, or using a Rule-based symbology using the category (field 5) as filter.
Linework is created by connecting points having same description and belonging to the same physical feature. *All line and polygon features are created as lines*.
The Initial Linework for the Survey Area is also stored temporarily in
+D: > AVSM > Zone-Survey number (eg RZ-01) > Survey Area (eg J) > Temporary WD+
h2. Note: The line shapefiles / Geopackages shall be in CRS: TM AUSPOS
h2. 2. Creation of final working drawing Shapefiles / Geopackages - (Selvarani, System 1)
Final working drawing Shapefiles / Geopackages are created from the Initial Linework of Survey Area.
As the Surveyor draws all features as lines (both for lines and polygons features), the following actions shall be done:
1. *If features are lines:*
• Export the shapefile / geopackage into the final working drawing folder (Final WD), in separate folders according to its type (e.g. BLDG, FENC, ROAD, etc).
h2. The CRS for the export shall be EPSG:4326 - WGS 84
2. *If features are polygons:*
• Lines shall be converted into polygons:
to do it, first click on the layer to be converted to make it active (e.g. WD-CZ-01-F-LL84_V-BLDG-MHOL------E), then go to “Vector” Menu, click on Geometry Tools, click on Line to Polygons:
!https://redmine.auroville.org.in/attachments/download/9760/Line%20to%20Polygon%20Menu.png!
The new window for “Lines to Polygons” conversion will appear:
!https://redmine.auroville.org.in/attachments/download/9762/Lines%20to%20Polygon%20Window.png!
• Always cross check the input layer, to make sure that the input layer is the active one
• Save the output in a temporary layer
• The temporary layer will be listed in the list of layers, it shall be exported to the saving location as +D: > Survey > Zone-Survey Number > Final WD > Survey Area SHP+ (eg . D: > Survey > GB-01 > Final WD > A-Shp)
h2. The CRS for the export shall be EPSG:4326 - WGS 84
Once all the shapefiles / geopackages are exported in Final WD, for each of the newly exported layers the Topology Checker Tool shall be used.
h2. Linework for the whole Survey Zone
h2. 1. Merging Shapefiles / Geopackages - (Selvarani, System 1)
A copy of the Zone Master shapefiles / geopackages are taken from System 4 and stored in Temp Folder on Desktop in System 1.
Master shapefiles / geopackages are merged with the Survey Area shapefiles / geopackages:
• To do it, go to “Vector” Menu, click on Geoprocessing Tools, then click on Union:
!https://redmine.auroville.org.in/attachments/download/9763/Union%20Menu.png!
The new window for “Union” will appear:
!https://redmine.auroville.org.in/attachments/download/9764/Union%20Window.png!
• To make sure that the right geometry is generated by this process (“line” type, not “Multiline”, and similarly “Polygon” type, not “Multipolygon), we need to always keep the *Master shapefile* (e.g. Final-CZ-01-2021-02-05-LL84_V-BLDG-MHOL------E) *as Input layer*, and the Survey Area shapefile as Overlay Layer (e.g. WD-CZ-01-F-LL84_V-BLDG-MHOL------E).
• (The output can be saved to a file, as the CRS should already be EPSG4326 – WGS84.)
h2. 2. Storing Shapefiles / Geopackages - (Selvarani, System 1)
Save the merged shape file in the correct location in Final folder as +D: > Survey > Zone-Survey Number > Final+ (eg . D: > Survey > GB-01 > Final)
Date in the name of Final Shapefile / Geopackage needs to be updated.
Once the merging operation is completed, the copy of Master shapefile / geopackage is deleted from the Temp folder.
h2. 3. Topology check of merged shapefiles
The topology checker is applied again on the merged shapefiles / geopackages.
The “id_field” shall be removed from the attribute table.
h2. 4. Archive and replace the Master Shapefiles / Geopackages (Ram, System 4)
Archive the previous master shapefiles / geopackages on system 4, and copy the new merged shapefiles / geopackages in its place.
*Then delete the Merged Shapefile / Geopackage folder from System 1.
*
h2. 5. Note about Shapefiles and Geopackages
All the above works are usually done using shapefile format, in QGIS latest version (3.16.3).
The Geopackage export is done in QGis versions older than 3.12 (e.g. 3.4, 3.6, 3.8, 3.10) so that the lines are not saved as “Multilines” but as “Lines”and polygons are not saved as “Multipolygons” but as “Polygons”. This is very important to be noted, as Gisaf database does not accept the Multipolygon and Multiline geometry types.
A different way to create "Polygons" is to use the command Vector -> Geometry Tools -> Multipart to Single Parts and apply it to the layer: from "Multipolygon" it will become "Polygon" (check in layer Properties).
As on 13 March 2021, Gisaf can accept Multipolygon layers, because the command "Multipart to Single parts" has been integrated into the importing command (see Redmine ticket #11691)
h1. Creating 3D Shapefile/Geopackage
While creating a Shapefile/Geopackage, the File name/Database-table name, Geometry type and CRS have to be entered.
In order to create a 3D Shapefile/Geopackage, the additional dimensions *"Z(+M values)"/ "Include Z dimension"* has to be ticked: this way the 3D Shapefile/Geopackage is accepted by Gisaf without errors, otherwise the Shapefile/Geopackage can't be imported in Gisaf because the Z dimension is missing (the geometries in the database are all 3D).
h1. Exporting from QGis (shapefiles and/or geopackages) to CAD dxf format
An algorithm has been created by Selvarani, to see the whole process click here:
https://redmine.auroville.org.in/projects/gis/wiki/Automatic_export_from_QGis_to_dxf
h1. [[CSR Geomatics Team]]
h1. [[Collaborations]]
h1. [[Data Analysis]]
h1. [[CSR Geomatics Server Setup]]
h1. [[Database Backup and Restoration]]
h1. [[CAD to GIS Workflow]] %{color:RED} - TO BE UPDATED%
h1. [[GIS Conventions]]
h1. [[Working with Gisaf]]
h1. [[Gisaf Workflow]]
h1. [[GIS to CAD Workflow]] %{color:RED} - TO BE UPDATED%
h1. [[Weather Station]]
h1. [[GDAL - Useful Tips]]
h1. [[Gisaf - Auxiliary Data (non-geographical data)]]
h1. [[QGis - Useful Tips]]
h1. [[AutoCAD Civil 3D - Useful Tips]]
h1. [[Wells documentation]]
h1. [[Gisaf Data Modifications]]
h1. Plan
Some interesting projects that might be integrated:
* https://github.com/Oslandia/albion : Build 3D geological model from wells information
h1. Links
h2. Water management
h3. Modflow
* https://water.usgs.gov/ogw/modflow/
The reference software for underground water modelling and simulation.
In conjunction with flopy (https://water.usgs.gov/ogw/flopy/) and Jupyter (https://jupyter.org/), it provides a _relatively easy_ to use interface.
h3. Freewat
* http://www.freewat.eu/project
This project is partly based on modflow, and integrates with QGis.
h2. QGIS
* https://www.qgis.org/en/site/
* Tools for Geology
Construction of geological cross sections in QGIS - http://www.geokincern.com/?p=1452
h2. Autocad
* Overview of Converting Geospatial Data to Drawing Objects:
http://docs.autodesk.com/CIV3D/2013/ENU/index.html?url=filesMAPC3D/GUID-C38FD485-3CC2-4B52-8264-0D8C0F45422B.htm,topicNumber=MAPC3Dd30e41809
* CAD-DB connection:
https://knowledge.autodesk.com/support/autocad-civil-3d/learn-explore/caas/video/youtube/watch-v-AQoB--nyUJA.html
h1. Orfeo
* https://www.orfeo-toolbox.org/
Remote sensing
h2. Reconciliation of points
The points are noted with their codes from and codes to reconcile into. ADD!! - *%{color:RED} to be written%*
h1. Survey Data Post-Processing
+Softwares used: Civil 3D, QGIS+
AutoCAD Civil 3D - C3D,
C3D is being used for post processing of survey data. C3D offers a BIM solution for Land Surveyors. Basic knowledge about C3D can be obtained by following tutorials online though a pre-acquired knowledge of Autodesk AutoCAD is a plus point and sufficient enough to start working with C3D.
QGIS is an Open source GIS.
Getting Started:
+Standards and Workflow for C3D+
The layers (Nomenclature of layers is coming from standards- U.S. National CAD Standard Version-AIA) are pre-assigned a geometry type (Point, Line and Polygon) in their layers description (Use layer manager to check). This is done keeping in mind inter-operability with GIS (In shapefiles format), geometry type Shape files are of three kinds- points, lines and polygons.
+Geometry types-+
Based on the description of Layers, if the layer Geometry is specified as 'Point', the raw survey data can be auto imported to webGIS - *GISAF*, no post processing is needed for 'Points'. For lines and polygons, We use Polylines (for 2D lines/Polygons) and 3Dpolylines (for 3D lines/polygons). In case of curved lines in 2D, Polylines command (PLINE/PL) is enough but its not possible to create curved lines in 3D using the 3Dpolylines, in such cases feature lines come into the picture. (*Important:* 2D polylines, splines, ellipses and circles are not exportable to shapefile format, they have to be always converted into Polylines).
*We are at the moment generating shapefiles (lines and polygons) in 2D (z=0) but keeping the format of files as 3D.
+Using Feature lines for creating Curved geometries in 3D-+
There are a series of steps involved when creating curved lines using feature lines. You can follow two methods depending on the situation :-
Method 1(For curved objects in 3D using elevation of points):- Create the object using 3DPOLY. Now use the create feature lines from objects and select the objects to be converted to feature lines. You will not notice any difference in the geometry on the screen as such but in properties you will see the object type description of selected object as feature lines.
Method 2(For curved objects in 3D by draping the lines/curves on a surface):- Create the object using PLINE. You will notice all the lines are straight. Now use the create feature lines from objects and select the objects to be converted to feature lines. You will not notice any difference in the geometry on the screen as such but in properties you will see the object type description of selected object as feature lines.
Now modify tab is used to convert the straight lines to curved lines. There are many ways to do it and different methods can be adopted in different situations. The easiest and the most commonly used method is by using the smooth command in the modify tab. After using the smooth command, a curve would be visible. This is a curve in 3D. To view it, object viewer option can be used by selecting the object and right clicking.
More complex methods have to be studied and each case has to be taken into consideration separatly when using - FITCURVEFEATURE In some cases to obtain a desired curve(Meaning more fragments) more PI's (Point of insertion) need to be added into the feature line. PI are the points where the feature lines gets fragmented when exploded.This step often becomes difficult to manage due to complexities involved in mathematical functions behind creation of feature lines.But it should be explored by all means.
Because the feature lines are not exportable to shapefiles, the feature line has to converted back into a 3D polyline. This can be done simply by explolding the object. Once exploded into 3D polylines, you will see that the curve gets fragmented into smaller segments of straight lines. You can repeat the process of creating the 3Dpolyline object into feature line and then smoothening it till you get a desired geometry resembling the curve.
Situations of sharing of points with different layers and proceeding with linework in such circumstances-
There is always a best way to take the survey points taken in the field by the surveyor. For instance, in case of a road and a curb adjacent to it, the surveyor takes the points only once. It is only during the post processing that the lines are generated (You can create the lines in field while taking the survey points as well, but that is not the most feasible method majorly due to time constraints) and in this case, two line on the same position shall be generated, one under layer of roads and one under layer of curbs.
The line work becomes complex in case of generating 3d lines and 2d lines on the same place. This can arise due to having some layers as 2D (for example building outlines) and some as 3D (paved surfaces- <taking into consideration the future use of the layers. In this case, for the purpose of water management, it becomes essential to have survey of such surfaces as 3D. Survey of building outlines doesn't need to be 3D because the elevation points are not recorded while surveying them due to feasibility).
So, in case of generating 3D linework from survey points with no elevation, Feature lines and SURFACES come into picture. The surfaces are generated by an interpolation method in C3D by giving a set of Point Groups. We have selected the triangulation method as the appropriate method due to availability of a dense set of points.
While creating the feature lines from objects (as explained above, check the box "assign elevation" and a dialog box appears to select the surface you want to select). Surfaces will have to be generated prior to using this. (add wiki for generating surface). Surface should be generated including the points of the layer in the point group of it.
*+Purpose of Survey+*
# The survey is conducted for generating a Base Map which is representative of Topography, Infrastructure, Drainage, Buildings etc, with a focus on Water Management in Auroville. Furthermore this survey can be used as base map to build upon more detailed surveys for the various purposes such as Town planning, Land Surveys, Transportation Planning etc.
+Use of Layers+
# *V-BLDG-SHED* (Shed) layer is used when the structure is not entirely enclosed, if At least one side is open (No walls/partition etc). One thumb rule is to answer the question if the building can be locked safely or not. If not, then it is a shed.
# *V-BLDG-HUT* (Hut) is used when the building (For human activity/living space) is single storey with roof made up of perishable material such as keet, straw etc.
# Use of *V-BLDG-ROOF* (Roof) is for buildings with permanent roofs extending out about more than 1 meter from the outline at ground level. It can also be used for complex roof structures for example Matrimandir Petals.
# Use of *V-BLDG-OTLN* (Building outline) is for depicting the outer line of walls of a structure on ground level. This is used when the roof of the building is more or less of the same profile.
# *V-BLDG-RTWL* (Retaining Wall) is for walls with the function of retaining either earth/water and the side of the wall should be visible. In case of very thin walls and no side visible, it comes as *V-BLDG-RWLL* (Retaining wall line). This is depicted as a single line in drawing.
# Use of *V-ROAD-CYCP* is for designated cycle path as a single line.
# Use of *V-ROAD-FPAT* is for pedestrian paths as a single line. It is used when there is a path of very small width. If the need is to represent a path/road which is unpaved in nature, *V-ROAD-UPVD* is used, which is polygon type. To represent a path/road of paved nature *V-ROAD-PAVD* is used, which is also a polygon type.
# *V-WMNG-PIPP* for pipe points, *V-WMNG-PIPL*- for pipe lines.
# *V-BLDG-BMRK* is for unidentified bench marks and *V-BLDG-MHOL* is for unidentified manholes.
# *V-BLDG-PLTF* is for impervious surfaces. for example sand and gravel will not come under this layer. cemented paths etc can come.
# *V-WMNG-POOL* is for water bodies made for leisure activities- like swimming pool etc.
# *V-WMNG-SUMP* is collectively for artificial water retention structures.
# *V-WATR-POND* is for top of natural water retention.
# *V-WATR-PONB* is for bottom of natural water retention.
# *V-WMNG-DRBA* is for bottom of artificial drains.
# *V-WMNG-DRTA* is for top of artificial drains.
# *V-WATR-DRBN* is for bottom of Natural drains.
# *V-WATR-DRTN* is for top of Natural drains.
# *V-WMNG-DRNC* is for drain covers.
# *V-WATR-DRAS* is for indicating slope of drainage, these are arrows.
h1. *Reconcilation of Raw survey data using pgAdmin*
+For changing the layer codes, to be done using Pgadmin.+
h3. Enter Schema> AVSM RAW Survey > Tables > select the table> Right click - View edit data > All rows. Apply filter to the original id and note the Database id's to be reconciled.
h3. GISAf Admin> Other > Reconciliation. Create > Add the database (point) id and give the new target corresponding to the database id to be changed. (Use the Layer name in target not Raw layer name).
h3. Other > Project > select the project > with selected> reconcile RAW survey points.
h1. Importing point data (TS and RTK) to GISAF
# *Gisaf Admin > Basket > Survey data > "Project" > "surveyor" > TS/ RTK > upload > import.*
# *Auto import of Raw points data* (changes from Raw to Shapefiles, the point files)
In GISAF Admin > Other> Project > select the project for which you want to import the data > with selected > Auto import to GIS...
h1. Editing Z value of features in Shapefiles in QGIS
h2. using vertex editor tool - https://www.youtube.com/watch?v=8V8i1AtcA74&t=256s
h1. Miscellaneous- Civil 3D
*Autodesk civil 3D Geotechnical module- for borehole data :- To analyse borehole data, To make profiles and calculate volumes*
https://www.youtube.com/watch?v=0Be9kShBou0
Reference videos
Exploring:
https://www.youtube.com/watch?v=gr-ISPzLcU0
Modelling:
https://www.youtube.com/watch?v=Pz0-HOoiBrs
solids
https://www.youtube.com/watch?v=mQ0Yeh6tZA8
https://www.youtube.com/watch?v=4eMsUiYBhuE
*CIVIL 3D Survey*
Getting started- Always open a new drawing with a template.
*Description key sets*- (till 18:30) https://www.youtube.com/watch?v=mmwkkRyBkS0
As when the points come into the drawing, they are going to be filtered with description key sets and the description key sets will assign properties to the points.
Tool space > settings > points > description key sets
“Description key sets name”> edit key- to view the points in a list:
Automatic linework (18:30- till end)
Survey > Linework code sets- for automatic linework
*Using Master view*
Uses-
copy styles from one drawing to other
copy drawing data from one drawing to other (data referencing)
*Civil 3d surface model*- from points- break lines
https://www.youtube.com/watch?v=wC40rdmDEMo
moving for example- point groups under point groups-
*Civil 3D Planning and Analysis*
Workspace: Planning and Analysis
1. Working with Object data (GIS Attributes)
Map Setup> Define object data> New table> Define new object data table start defining fields
h1. Documentation- Rain Gauge
h2. Manual Rain Gauge
*Why it is important to comply to standards?*
1. Consistency for comparing rainfall data in different places within Auroville since rainfall varies in different parts of Auroville.
2. Making it possible to use for any kind of scientific analysis by contribution to the primary data in sustainable water management for Auroville.
*Proposed standards and ethics:*
1. The time of taking the reading - 08:30 am.
The rain is recorded over a period of 24 hours- 8:30 of previous day till 8:30 of the present day, and the date is put as the present day. For example, if the rain is measured at 8:30 am on 10 dec 2017, then it is logged in as on 10 dec 2017.
2. Manual rain gauge typically used- green cylindrical and a collection jar.
3. Measuring jar: 10 mm corresponding jar
Area of the rim of manual rain gauge= 200 square cm
Diameter of rim= approximately 16 cm
*NOTE:* - If another kind of rain gauge is used, it should be used with the corresponding measuring jar as type of jar and calibration on jar depends on the area of rim. Any cases of different rain gauge other than above should be reported before use for recording purposes.
4. The minimum record-able unit is 0.2 mm (Least measure on the 10 mm cylinder). T is marked for below 1 mm.
5. The reading should be preferably submitted in an excel sheet (the data sheet will be provided). The frequency of sending data can be daily or weekly.
6. The rain gauge should be kept in a safe and open to sky area clear from tree cover or objects in surrounding so that direct rain falls into it and not from trees/ objects etc. Preferably roof top or a higher place on ground.
7. A check on the rain gauge should be kept when there is long period of gap in rainy days to make sure it was emptied out and there is no blockage.
8. In case of doubt in reading or incorrect measure, "incorrect measure" should be marked
*Ethics*
1. Readings should not be missed. Time to time check on rain gauge is required during long periods of no rain (especially when the rain season is approaching).
This is important because firstly, if it rained in the night and it wasn’t recorded, reading for the day is lost; Secondly, if the jar was not emptied, it gives incorrect subsequent reading.
2. If the person is not going to be available temporarily where the rain gauge is kept, the responsibility should be passed on to someone they can rely on after explaining the standards.
*Important-*
Data won’t be published in case of non-compliance to standards. This is to maintain the sanctity of proper scientific data collection and to keep it reliable for sharing by publishing. Discussions regarding standards are welcomed.
CSR Geomatics Team is placed on first floor in CSR, Auroshilpam.
*AV rain data publication*: (http://gis.auroville.org.in/measures/raingauge_av).
Adding Rain gauge to our web portal (GISAF), following information is needed:
1. GPS co-ordinates of location of placing the rain gauge (can be obtained using mobile phones easily)
2. Name, place of residence & contact number- Mobile & Landline
h2. Automatic Rain Gauge
There is a possibility of publishing rain data coming from Automatic rain gauges like (id 15, auro orchard) and Weather stations.
For Automatic rain gauges, we can upload the files coming from the rain gauge directly into GISAF.
_Thank you for contribution towards a sustainable management of Water in Auroville through Data collection._
h1. Documentation- Wells Monitoring (Manual and Automatic / Piezometer by Bala)
h2. Manual- Using tape with a sensor
Timings: The person assigned the job of monitoring collects the readings in three slots.
1. Between 6 am to 7 am till about 9 am to 10 am.
2. Between 11 am and 1 pm
3. Between 2 pm and 5 pm
Things required:
• Notebook
• Pen
• Measuring tape
• Vehicle for movement
*Ethics*-
• The monitoring should happen in coordination with the community members/ care taker (assigned by the person in charge from within the community). There should be a clear communication from monitor side as to which days the monitoring of a well has to happen and at around what time so that it can be made sure that the pump is not turned on before monitoring. In case a pump was turned on in a well, the monitor should have a gap of about 5 hours on the same day before going again for monitoring.
• The monitor should have contact numbers of the person in charge/ care taker for any communication.
• If the monitoring is stopped for any reason at any point, the monitor should communicate the same to the person concerned.
• The monitor shall take responsibility to inform any kind of changes in a well in terms of its functioning etc.
• The monitor should report to the CSR Geomatics Team who has responsibility to publish data.
• The monitor published the data to the website and works with the geomatics team.
*Definitions*:
A master file is maintained with the records of the wells from the field. It is updated when a new well is located. The following set of information are filled out in the Masterfile.
*Well type*
1. Open
2. Borewell
3. Dug cum borewell
*Well status*
1. In use
2. Not in use
3. Closed
*Pump Status*
1. Functioning
2. Not functioning
*Pump Automation*
1. Manual
2. Automatic
* Non-accessibility factors*
1. Dogs
2. Heavy slab
3. Narrow casing
4. enclosed/locked
5. permission
*Data Matching Accuracy (reconnecting with harvest wells data)*
1. High
2. Medium
3. Low
*Well coordinates*
For a new well, coordinates of the well position are taken on a mobile GPS.
Person In charge
For communication purpose, contact the person in charge as recorded in directory.
h2. Automatic- Piezometer (by bala, to be edited)
The calibrations and setting are already done by Azha
The piezometer is taken to the site of unused well. First the depth and Water level is checked manually using water meter
Then the sensor of the piezometer is inserted into the well 1m above the bottom of the well, so the sensor does not get affected from mud or water inside
The sensor sends the data to an electronic board which translates the signal into the proper output, and then it sends the signal to the transmitting device above, on the ground surface. The transmitter then sends it to the receiver at CSR, the signal is then routed to Talam office through Interneet, enters the software and the reading is processed.
The piezometer sometimes does not have proper signal and so it would not be able to send it
h1. Documentation- Flow meter by Bala
Flow meter – for checking the flow of water in pipe
CSR bought ultrasonic flow meter from Chennai and the company (company name? )trained Bala and Vijai(CSR) on how to set it up.
we tested in the west water system pump in csr. then we test in many more place .
ami, aurodam and buddha garden borewell pumps . we got request from the water service
Cross check they flow meters .
so before that we want to know how our meter works.
so we tested in our tank . the pump pumping from our sump tank so the flow will be stranded. when you pump in the bore well the flow goes up and down.
we tested two times in 15 minutes.and one time 30 minutes,so we got variation between this three.
we find 1,5 % error but the flow meret they said 1 % error only.
we cross checked the flow meter of the water service we find some error in they meter also.
we tried they bore well flowmeter and they sump tank flow meter also.and we find some error also in they meters.
then we find some error in (AVWS) meter also.
It is used to check the flow of water in pipe
tools used
First the outer diameter of the pipe has to be entered in the device. It is measured using Vernier caliper
Then the thickness of the pipe is set which is also measured using vernier
Also the device asks for the material of the pipe. If the pipe material is know it can be set and if it not know then there is an option which is other (mostly PVC, HDPE, and iron)
After entering these details the device gives the spacing for the sensors. There are 2 sensors up and down which has to be fixed accordingly
When the sensors are fixed the motor is turned on and the sensors send reading to the display device
This is noted down once every minute and taken for 15 to 30 minutes. Then the average of this is determined. This is done because of the variation in the flow. This gives the flow rate in 1 hour.
This was done in different places to check to flow rate
h1. Documentation- DST- Vegetation Indexing
h2. Steps for Dzetsaka Classification tool for Vegetation indexing in QGIS
1. Install the plugin Dzetsaka classfication tool.
2. Open the Raster from the Survey.
3. Create a polygon shapefile for index sampling. Mark polygons and give the ID's (1,2,3 for Tree, grass, bare land etc) Cover the variations in samples as much as possible.
More the samples, better the indexing.
4. Apply Dzetsaka Classification tool, Select the base raster and the sample- index polygon shapefile created in step 3.
5. The result is a Raster with DN numbers specified in the Shapefile in step 3.
6. Apply the Sieve raster command (Raster> Analysis> Sieve)- Try different threshold numbers and view the results till the noise is removed from the Raster.
7. Polygonise the Raster to Vector (From processing)
8. Run the v.generalise tool on the shapefile. This tool removes the pixelated boundaries of the polygons in the Vector.
h1. Documentation- DST- Interpolation (Processing toolbox)
The following 4 tools have been mostly explored and the results were compared. The ones used for quick analysis are 1. Cubic Spline and 2. V.surf.spline . The rest of the tools are for further exploration and used depending on the need of the project. In some tools, the elevation values of points should be stored in the attribute table (Using field calculator and giving command - *Z($Geometry)* )
# Interpolate (Cubic spline) - SAGA
# V.surf.bspline - GRASS. Parameters to set - cell size. Set this parameter above 0.001 ( 0.00001, 0.000001 etc) and check the results.
# V.surf.rst - GRASS
# Krigging - SAGA
[[Documentation- DST- Survey- Office workflow]] - *%{color:RED} to be written%*
h1. From CAD to GIS by Giulio
*FEATURES CREATION*
1. Assign a CRS to the drawing (TM-AUSPOS) (MAPCSLIBRARY command)
2. Create features in CAD (Points, lines, polygons)
3. Export shapefile (a) from CAD (Output > DWG to SDF) (Convert to LL84 – 3D)
*FEATURES IMPORT INTO DB FIRST TIME*
4. Create zip file of the shapefile
5. Upload into the GISAF Shapefiles Basket
6. Import the shapefile into DB
7. Save the shapefile on Local Machine
*FEATURES IMPORT INTO DB EVERYTIME*
8. Combine the new features to corresponding last shape files (Insert the process here).
9. Follow step 4-8 again
*FEATURES EDITING IN QGIS*
10. Open the table in QGis
11. Save as a shapefile (b) in TM AUSPOS CRS
12. In CAD, open a new drawing and assign AUSPOS CRS
13. Import the shapefile (b) (MapImport) with Object Data (Data tab > Create Object Data > OK), tick “Import polygons as closed polylines”, then press OK
14. Edit features
15. Change workspace into “Planning and analysis”
16. Export shapefile (a) from CAD (Output > DWG to SDF) with ONLY the id selected (Data Tab > Select Attributes > Object Data > Filename > id) (Convert to LL84 – 3D)
*FEATURES IMPORT INTO DB*
17. Create zip file of the shapefile
18. Upload into the GISAF Shapefiles Basket
19. Import the shapefile into DB
20. Delete the shapefile from Local Machine
h1. QGIS- Miscellaneous
*QGIS Introduction:*
https://www.birdseyeviewgis.com/blog/2018/2/22/my-favorite-features-of-qgis-30to-date
3D visualization of raster DEM- https://www.youtube.com/watch?v=2KrCsbP0kUs
*Spatial query:*
Spatial Query is selection of features that satisfies a certain condition which relates to other features in a space.
Using plugin- Spatial query
http://www.geodose.com/2018/03/spatial-query-in-qgis-3.html
Labelling:
Labelling with more than one field names and in different lines
|| '\n' ||
*Hierarchy of extensions*
File levels and their uses.
https://nathanw.net/2014/03/22/all-the-q-files/
+The Project file (.qgs)+
It contains: Layer source pointer + Style information + Composers + a whole heap of other stuff
+The Layer Definition file (.qlr)+
It contains: Layer source pointer + Style information
+The QML file (.qml)+
It contains: Style information
h2. Giulio's documentation
h1. Documentation - Reconciliation of points using Gisaf
Reconciliation of points is a procedure used when a point is stored in a wrong table, because its category was either wrongly recorded in the field by the surveyor, or it has ben reviewed later by the surveyor or the data validator and found to be wrong.
h2. Definition: Raw points are all points coming from the field survey. Raw points can be points referring to a Point feature (e.g. trees, or elevation points, or floor level), or points measured in the field to draw a line (e.g. the vertices of a fence) or a polygon (e.g. the corners of a building outline).
In the overall workflow, *+reconciliation affects only raw points+*. It takes place after the field textfile is uploaded into the basket and its points imported (raw points stored in the raw survey tables). Here you can visualize the Workflow diagram: [[Survey_data]].
If a raw point refers to a point feature, reconciling it means moving it to another category/table meant for point features, not for lines/polygons. So, *+only raw points of point features can be reconciled+*. Raw points pertaining to line features and/or polygon features cannot be reconciled, so these raw points will remain in their original wrong table.
PLEASE NOTE:
The attributes of a line/polygon, which are derived from their defining raw points, will not be modified by any reconciliation, because *+only raw points of point features can be reconciled+*.
h2. How to perform Reconciliation
To perform reconciliation of points: Login to Gisaf -> click on the G icon on the upper-left corner of the website page -> Manage -> Reconciliation by orig.ID
On the right end side of the screen, clicking on the field "Destination" a list of all categories will appear: these categories refer not to the Raw survey tables, but to the V_ tables of the database (points, lines, polygons).
Under it, in the field "Original ID", the original point number of the point to be reconciled is to be entered.
Clicking on the "Search points" button, the result shows the database unique id of the point, its survey category, its survey date, its geometry type (point, line, polygon), and the Project the point belongs to. In case of multiple points with the same original id (in case of different Projects, the field number of points might be not unique if the numbering of points in the field has restarted) all points having that original id are displayed: thanks to their date or Project or type, it is easy to identify the correct point to be reconciled.
Once the point to be reconciled is identified, clicking on the button "Reconcile" will run the reconciliation, and a message will appear stating that it has been done successfully. An error message can appear if a reconciliation of a raw point of a line/polygon feature has been attempted: this type of points cannot in fact be reconciled.
Once a raw point has been reconciled, *+it cannot be reconciled again+*. In case a raw point has been wrongly reconciled, it cannot be reconciled again through the above procedure, but it has to be reconciled manually through QGis or pgadmin software.
h1. Documentation - Status and Status Changes
Status have been created to keep track og changes in surveyed features.
It is an additional (though provided for in AIA standards), single digit value, at the end of the Gisaf Category/Cad layer name.
Each Status need to have a corresponding CAD layer/Gisaf Category, with a short code associatedto it, so that field entries can be done easily.
Status have been defined as follows:
N - New Work
E - Existing o remain
D - Existing to demolish, Demolished or Changed
F- Future work, Proposed feature
T - Temporary work
M - Item to be moved
X - Not in contract
By default, Status is defined as E (Existing). Status can anyhow be changed later using gAdmin, or QGis (through the PostGis connection).
It needs to be done manually, one feature (point, line, polygon) at a time.
In the future Status changes might be incorporated in the Admin panel.
h1. Documentation - Tags retained after re-import of same geometry
Documentation - Tags retained after re-import of same geometry
The linework for infrastructure survey carried out by Eric Chacra in May 2020 was originally imported with a problem of ambiguity in the "Accuracy" table and in the "Accuracy" table.
The result was that lines did not inherit the attributes survey date, accuracy, equipment, surveyor.
Nevertheless lines were displayed on the Gisaf map, without these attributes, and tags were given to some of these lines.
The values for the two tables ("Accuracy", "Accuracy") have been corrected, ambiguity resolved.
The lines in the layers V-ELEC-UGND------E and V-COMM-CABL------E have been then reimported, the attributes have been properly assigned, and the tags have been retained.
4 August 2020
h1. Access to data
h2. Connection to server directly from CSR
To connect to the server directly without going through Aurinoco server, the correct url is
http://gis.csr.av
h2. Connection to Gisaf via QGis through WFS / OGC API
This works only on QGis from version 3.14.15 onward
In the browser, click on WFS/OGC API, then right-click to create a new connection
Give a name (e.g. OGC API Qgis Gisaf)
Give the url https://gis.auroville.org.in/ogcapi
Under the WFS Options box, on Version dropdown, the default option "Maximum" works just fine
Click on OK
The list of layers will appear in the Browser under WFS/OGC API.
h1. How to create a new projection in QGis
To create a new projection in QGis, go to menu "Settings", and click on "Custom Projections".
A pop-up window appears with a list of all projections defined in QGis projects used by the user so far.
Click on the green "+" sign on the right top part of the window to create a new projection.
In the "Name" box, type "TM CSRAUSPOS SF1" (which means TM = Transverse Mercator projection; CSRAUSPOS = theparameters for this projection are derived from the processing of DGPS raw data by AUSPOS - Online GPS Processing Service - https://www.ga.gov.au/scientific-topics/positioning-navigation/geodesy/auspos; SF1 = Scale Factor is 1).
In the "Format" dropdown list, select "Proj String (legacy - Not Recommended)"
In the "Parameters" box, paste the following "+proj=tmerc +lat_0=12.01605433+lon_0=79.80998934 +k=1 +x_0=370455.630 +y_0=1328608.994 +ellps=WGS84+towgs84=0,0,0,0,0,0,0 +units=m +no_defs".
Finally, click on OK.
In a more explicit way, the parameters mean the following:
Map Projection: TransverseMercator (TM)
False Easting: 370455.6300
False Northing: 1328608.9940
Latitude of Origin: 12°00'57.79560" (DMS) 12.01605433 (DD)
Central Meridian: 79°48'35.96164" (DMS) 79.80998934 (DD)
Scale Factor: 1.00000000
Zone Width: 6.0°
h1. Elimination of Duplicate points – General criteria
It might happen that the same physical feature (e.g. a tree, or a pole) is surveyed more than once: this can happen because there are many physical features in an area, and the survey needs more than one station. So, for example a tree is surveyed from a station, and gets a serial number on that date. When the station is then changed, it might happen that the same tree is resurveyed: another serial number is given, and possibly a different date, if the survey from the second station happened on a different day.
It is clear that the same tree is then represented with two different points, which means that two different trees exist: but only one tree really exist in the physical reality.
It is clear that one of the two points is redundant and needs to be removed. If this is noted by the surveyor directly in the field, then the issue is solved by the surveyor himself during processing time.
If instead, due to various reasons, it was not noted by the surveyor in the field, it will need to be cleaned after the processing, possibly by post-processing staff.
How to identify duplicate points?
The following criteria can be used:
1. The distance between the two points is less than 30 cm (trees are surveyed if their trunk diameter is at least about 20 cm, so in 30 cm cannot exist two of them)
2. The orig_id (serial number) of the points are not in series
3. The survey date is not the same
4. In case of trees, the species of trees is the same
5. 5. In case of trees, the tree type is not TDEF (because TDEF are mapped irrespective of their diameter, so they can actually have a small trunk, and two of them might exist in 30 cm), not OT (many TDEF species are surveyed as OT if not otherwise indicated by a botanist)
6. The context needs to be evaluated: if one tree is deleted in an area where many trees exist in a limited space, then loosing one in the map is not a big error. If instead one tree is deleted where there are very few trees, then it might be a big loss.
h1. Linework for the Survey Area
h2. 1. Creation of Initial Linework in QGIS using Survey points import - (Ram, System 4)
Initial Linework in QGIS is started by surveyor with the knowledge from the Field. For this step, points are simply imported into the QGIS from the field text file (.csv or .txt). CRS needs to be TM-AUSPOS. The box of “First record has field names” shall not be ticked. In Point Coordinates, select the correct field for x, for y and for z (usually “field_2” for x, “field_3” for y and “field_4” for z). Points can be styled using the “Categorized” style in “Symbology”, using “Field_5” as value, or using a Rule-based symbology using the category (field 5) as filter.
Linework is created by connecting points having same description and belonging to the same physical feature. *All line and polygon features are created as lines*.
The Initial Linework for the Survey Area is also stored temporarily in
+D: > AVSM > Zone-Survey number (eg RZ-01) > Survey Area (eg J) > Temporary WD+
h2. Note: The line shapefiles / Geopackages shall be in CRS: TM AUSPOS
h2. 2. Creation of final working drawing Shapefiles / Geopackages - (Selvarani, System 1)
Final working drawing Shapefiles / Geopackages are created from the Initial Linework of Survey Area.
As the Surveyor draws all features as lines (both for lines and polygons features), the following actions shall be done:
1. *If features are lines:*
• Export the shapefile / geopackage into the final working drawing folder (Final WD), in separate folders according to its type (e.g. BLDG, FENC, ROAD, etc).
h2. The CRS for the export shall be EPSG:4326 - WGS 84
2. *If features are polygons:*
• Lines shall be converted into polygons:
to do it, first click on the layer to be converted to make it active (e.g. WD-CZ-01-F-LL84_V-BLDG-MHOL------E), then go to “Vector” Menu, click on Geometry Tools, click on Line to Polygons:
!https://redmine.auroville.org.in/attachments/download/9760/Line%20to%20Polygon%20Menu.png!
The new window for “Lines to Polygons” conversion will appear:
!https://redmine.auroville.org.in/attachments/download/9762/Lines%20to%20Polygon%20Window.png!
• Always cross check the input layer, to make sure that the input layer is the active one
• Save the output in a temporary layer
• The temporary layer will be listed in the list of layers, it shall be exported to the saving location as +D: > Survey > Zone-Survey Number > Final WD > Survey Area SHP+ (eg . D: > Survey > GB-01 > Final WD > A-Shp)
h2. The CRS for the export shall be EPSG:4326 - WGS 84
Once all the shapefiles / geopackages are exported in Final WD, for each of the newly exported layers the Topology Checker Tool shall be used.
h2. Linework for the whole Survey Zone
h2. 1. Merging Shapefiles / Geopackages - (Selvarani, System 1)
A copy of the Zone Master shapefiles / geopackages are taken from System 4 and stored in Temp Folder on Desktop in System 1.
Master shapefiles / geopackages are merged with the Survey Area shapefiles / geopackages:
• To do it, go to “Vector” Menu, click on Geoprocessing Tools, then click on Union:
!https://redmine.auroville.org.in/attachments/download/9763/Union%20Menu.png!
The new window for “Union” will appear:
!https://redmine.auroville.org.in/attachments/download/9764/Union%20Window.png!
• To make sure that the right geometry is generated by this process (“line” type, not “Multiline”, and similarly “Polygon” type, not “Multipolygon), we need to always keep the *Master shapefile* (e.g. Final-CZ-01-2021-02-05-LL84_V-BLDG-MHOL------E) *as Input layer*, and the Survey Area shapefile as Overlay Layer (e.g. WD-CZ-01-F-LL84_V-BLDG-MHOL------E).
• (The output can be saved to a file, as the CRS should already be EPSG4326 – WGS84.)
h2. 2. Storing Shapefiles / Geopackages - (Selvarani, System 1)
Save the merged shape file in the correct location in Final folder as +D: > Survey > Zone-Survey Number > Final+ (eg . D: > Survey > GB-01 > Final)
Date in the name of Final Shapefile / Geopackage needs to be updated.
Once the merging operation is completed, the copy of Master shapefile / geopackage is deleted from the Temp folder.
h2. 3. Topology check of merged shapefiles
The topology checker is applied again on the merged shapefiles / geopackages.
The “id_field” shall be removed from the attribute table.
h2. 4. Archive and replace the Master Shapefiles / Geopackages (Ram, System 4)
Archive the previous master shapefiles / geopackages on system 4, and copy the new merged shapefiles / geopackages in its place.
*Then delete the Merged Shapefile / Geopackage folder from System 1.
*
h2. 5. Note about Shapefiles and Geopackages
All the above works are usually done using shapefile format, in QGIS latest version (3.16.3).
The Geopackage export is done in QGis versions older than 3.12 (e.g. 3.4, 3.6, 3.8, 3.10) so that the lines are not saved as “Multilines” but as “Lines”and polygons are not saved as “Multipolygons” but as “Polygons”. This is very important to be noted, as Gisaf database does not accept the Multipolygon and Multiline geometry types.
A different way to create "Polygons" is to use the command Vector -> Geometry Tools -> Multipart to Single Parts and apply it to the layer: from "Multipolygon" it will become "Polygon" (check in layer Properties).
As on 13 March 2021, Gisaf can accept Multipolygon layers, because the command "Multipart to Single parts" has been integrated into the importing command (see Redmine ticket #11691)
h1. Creating 3D Shapefile/Geopackage
While creating a Shapefile/Geopackage, the File name/Database-table name, Geometry type and CRS have to be entered.
In order to create a 3D Shapefile/Geopackage, the additional dimensions *"Z(+M values)"/ "Include Z dimension"* has to be ticked: this way the 3D Shapefile/Geopackage is accepted by Gisaf without errors, otherwise the Shapefile/Geopackage can't be imported in Gisaf because the Z dimension is missing (the geometries in the database are all 3D).
h1. Exporting from QGis (shapefiles and/or geopackages) to CAD dxf format
An algorithm has been created by Selvarani, to see the whole process click here:
https://redmine.auroville.org.in/projects/gis/wiki/Automatic_export_from_QGis_to_dxf