GIS

h1. %{color:BLUE}  Wiki%

h1. [[CSR Geomatics Team]]

h1. [[Collaborations]]

h1. [[Data_analysis]]

h1. Our workflow

h2. Surveys

Most field surveys are executed by our team of surveyors, using advanced DGPS equipment.

Other surveyors might also be contracted.

h2. CAD

The survey data are imported to a CAD software (Aurocad/Civil 3D).

h2. Editing shapefiles in Autocad

h2. FEATURES CREATION

1.	Assign a CRS to the drawing (TM-AUSPOS) (MAPCSLIBRARY command)

2.	Create features in CAD (Points, lines, polygons)

3.	Export shapefile (a) from CAD (Output > DWG to SDF) (Convert to LL84 – 3D)

h2. FEATURES IMPORT INTO DB

4.	Create zip file of the shapefile

5.	Upload into the GISAF Shapefiles Basket

6.	Import the shapefile into DB

7.	Delete the shapefile from Local Machine

h2. FEATURES EDITING

8.	Open the table in QGis

9.	Save as a shapefile (b) in TM AUSPOS CRS

10.	In CAD, open a new drawing and assign ASUPOS CRS

11.	Import the shapefile (b) (MapImport) with all Object Data

12.	Edit features

13.	Export shapefile (a) from CAD (Output > DWG to SDF) with ONLY the id selected (Data Tab > Select Attributes > Object Data > Filename > id) (Convert to LL84 – 3D)

h2. FEATURES IMPORT INTO DB

14.	Create zip file of the shapefile

15.	Upload into the GISAF Shapefiles Basket

16.	Import the shapefile into DB

17.	Delete the shapefile from Local Machine

h2. QGis

h3. Conventions

h1. Shapefiles

We work with "QGis":https://en.wikipedia.org/wiki/QGIS , a widely used open source, free software for working on maps and geographical data.

"Shapefile":https://en.wikipedia.org/wiki/Shapefile is a standard file format for mapping, that Gisaf can import. QGis can open and save *shapefiles*.

We have defined some simple norms about these shapefiles for integration with Gisaf.

h2. Coordinate system

We use CRS SRID 32644.

h2. Column (attributes)

* All objects in a shapefile (layer) have a unique identifier named "id": numerical value.

h3. Field (attribute) names

* All fields are lower case (-UpperCase-, use: @lowercase@)

* They do not contain spaces, but underscores (-not this-, use: @but_that@)

* The field names cannot have more than 8 characters (-this_is_too_long-, use: @this_short@)

h3. Attribute types

* The dates are written in ISO format: @YYYY-MM-DD@ (eg. @1968-02-25@)

h3. Fields to remove

* Eventually, remove the fields containing the coordinates: northing, easting, elevation, latitude, longitude, area, length, etc (these might be present when the data is imported from speadsheet)

h2. Foreign keys

We often deal with labels or categories of objects in a layer.

A common use case to explain: Alice creates a layer of stones, and wants to *tag* each stone with a rating: these are picked from a list of choices, like: _Beautiful_, _Interesting_, _Pretty_, etc.

For these kind of attribute: define a column like @rating_id@ (_something_ ending with @_id@) as a numerical value, and assign values 1, 2, 3, etc. The text is defined in another table (typically a CSV file), that looks like:

|_. id |_. name |

| 1 | Beautiful |

| 2 | Interesting |

| 3 | Pretty |

h2. Code

We have defined a standard set of codes, that defines the type of data. They can be found here (TODO: add link).

Add a column @code_name@, matching with one the code, eg. @V25@ for TDEF.

h2. Surveyor

We keep a record of the people who realized the surveys (the _surveyors_).

The shapefiles must contain an attribute @srvyr_id@, which refers to this table (TODO: add link).

h2. Accuracy

We keep a record of the accuracy of the surveys.

The shapefiles must contain an attribute @accur_id@, which refers to this table (TODO: add link).

h2. Date of survey

As nothing is absolutely permanent, it's also important to keep track of the date of the surveys: the shapefiles must contain an attribute @date@.

h1. Working with Gisaf

h3. Survey data

Raw survey data are contained in CSV files, typically downloaded from surveying instruments.

See more information of the process for the survey data (including a flow diagram)

h1. Survey data

h2. Workflow summary

p=. !https://redmine.auroville.org.in/attachments/download/9839/Survey_data.jpg!

h2. Import basket

Gisaf provides an "import basket" dedicated for raw survey data, which is generated by land survey equipment (Leica's Total Station and RTK). 

These are CSV files, like:

<pre>

100081,370633.969,1327742.157,51.187,,,

100083,370628.876,1327702.913,51.565,T52,,

100082,370628.729,1327720.019,51.261,T52,,

100081,370633.969,1327742.154,51.179,,,

100083,370628.876,1327702.913,51.565,T52,,

20800,370633.969,1327742.154,51.180,,,

20801,370618.795,1327713.172,52.817,E30,,

20802,370623.674,1327711.436,51.283,B35,,

20803,370619.314,1327713.407,51.383,B35,,

</pre>

Each category (5th column) must be defined in the Category table (see [[Categories]]).

h2. Organization of the raw survey data basket

The basket should be organized in a directory structure:

- Project name (these can be themselves put in a hierarchy of (sub)directories)

- Surveyor's organization

- Equipment (eg. TS, RTK)

- Survey files (eg. @Our_project-Some_comment-2018-02-23.txt@)

h3. Format of the survey file names

<pre>

Our_project-Some_comment-2018-02-23.txt

</pre>

The date of the survey follows the ISO date standard: @YYYY-MM-DD@.

h2. Import to the database

When importing raw survey data files to the database, Gisaf does 2 steps as described below.

It's worth noting that, in this process, no reprojection is done.

h3. Feed the raw_survey table

Each point of the imported raw survey data file is inserted to the raw_survey table:

# Creation of a Point geometry: the raw_survey table has a geometry column for a single point (@geom@) with x,y and z coordinates

# Save the @id@ of the original point *to the @orig_id@ column*

# *A unique @id@ is computed* from the following fields: @id@, @project@, @equipment@, @date@

# The project is saved in the @project_id@ column

# The surveyor identification in @srvyr_id@

# The date of survey is saved in the @date@ column

# The accuracy is tagged in the @accur_id@, according to a mapping defined in the @accuracyequimentsurveyormapping@ table, which depends on the surveyor and equipment

# The category of the point

h3. Feed the @RAW_V_*@ tables

From the @raw_survey@ table, each point is then copied to its respective @RAW_V_@ table, with basically the same information.

These tables (which should be created manually or with the admin notebook called @create_tables@, as of today's writing), only contain points.

The project is saved along: see below.

h2. Import the points

For categories that define points (opposite to lines and polygons, which require _line work_ carried over in CAD or in a GIS software, see [[Line work]]), the points can be imported automatically to their final destination: the @V_*@ tables.

Note: in this process, the geometries are reprojected.

h2. Auto import of the points

The points found in the @RAW_V_*@ tables can be imported automatically, project per project, the project page of the admin interface.

h2. Import of the line work (lines and polygons)

h2. See [[Line work]] - *%{color:RED} to be written%*

The shapefiles generated manually (line work) should be put in the project's basket, and imported from it.

h1. Categories

The categories define the types of the geographical features and they are mapped according to ISO standard layer naming conventions: see https://www.nationalcadstandard.org/ncs5/pdfs/ncs5_clg_lnf.pdf

Gisaf uses:

* a table @category@ where the layers are defined

* a table per category

h2. Fields for the categories

*%{color:RED} to be written%* - TODO

h2. Creation of the RAW_* tables

This step must be done manually (as of today's date of writing).

h1. QGis: work on shapefiles

h2. Go to [[shapefiles]] - *%{color:RED} to be written%*

h1. Data an

We use "Jupyter":https://jupyter.org , "Pandas":https://pandas.pydata.org/ and "GeoPandas":http://geopandas.org/ , accessible at http://gis.auroville.org.in/notebooks .

For integration in the processes (execution of notebooks), there's "papermill":https://github.com/nteract/papermill . Systemd "timers":https://wiki.archlinux.org/index.php/Systemd/Timers are used to automatically schedule the notebooks on the server, ie. for the dashboards.

There's a dedicated virtual machine for Jupyter, accessible from our local network at @jupyter.csr.av@.

h2. Organization of notebooks

The setup is organized in 2 parts, that are run with 2 instances of Jupyter for security reasons.

h3. *Admin*

The notebooks in the admin are mostly for maintenance: operations on the database, etc.

h3. *Users*

The notebooks are organized in folders, all under Gisaf's source code git repository, except the "Sandbox" one.

This notebook server connects to the database with a specific user (@jupyter@), which has been set on the database server with permissions to read all data (@readonly@) plus has write access to some tables dedicated to store analysis results.

h2. Integration with Gisaf

The notebook in @Templates@ demonstrates the usage of notebook in relation with Gisaf: mostly, how to use the @gisad.ipynb_tools@ module to access Gisaf models and the data from the database.

This module is part of gisaf: https://redmine.auroville.org.in/projects/gisaf/repository/revisions/master/entry/gisaf/ipynb_tools.py

h2. References

h3. Geopandas

Some nice examples of processing, using water shed and rain: https://geohackweek.github.io/vector/06-geopandas-advanced/

h3. Integration

A good example of how a company has integrated the same tools: https://medium.com/netflix-techblog/scheduling-notebooks-348e6c14cfd6

h2. Other docs

h3. *%{color:RED} From Swathi%*

h1. Hosting

The team is located in the CSR of Auroville.

We have setup a server for hosting the software and database. *CSR_server* for technical information about the setup.

h1. CSR server

dream.csr.av (192.168.0.12)

- Debian 9

- Xen hypervisor

- libvirt for the orchestration of VMs

- management with ansible

h2. Dom0

h3. Installation

Found minor issues with the installation (eg. issues with HDDs, Dell EFI boot).

This document starts from a working Debian Xen server.

-Installed on a mirror of 2*2TB drives with btrfs.-

Update (see #7156): re-installed the OS on /dev/sdc2, ext4, without RAID/LVM.

h3. Storage for domUs

A LVM dream.csr has been created for the domUs.

h3. Networking

With systemd.networkd: bridge "br0" to the physical ethernet.

h3. Create a VM

Adjust the parameters from:

<pre>

export vm_name=infra.csr.av

export vm_ip_addr=172.16.0.3

export vm_password=foobar

</pre>

Create the domU:

<pre>

root@dream:~# xen-create-image --broadcast=172.16.0.255 --dist=stretch --fs=ext4 --gateway=172.16.0.1 --hostname ${vm_name} --ip=${vm_ip_addr} --lvm=dream.csr --maxmem=512M --memory=256M --mirror=http://ftp.de.debian.org/debian/ --netmask=255.255.255.0 --password=${vm_password} --size=10G --swap=1G --vcpus=1 --bridge=br0

</pre>

Note that IP address will be set in the VM, and the vm_ip_addr isn't actually used.

h2. DomUs

h3. Migrate XL to libvirt

After creation using xen-create-image, migrate the definition of the domU to libvirt:

<pre>

virsh -c xen:/// domxml-from-native xen-xm /etc/xen/${vm_name}.cfg > /tmp/${vm_name}.xml

virsh define /tmp/${vm_name}.xml

</pre>

From this point onward, one can log out from the dom0's console and use virsh or "Virtual Machine Manager" from your computer to administer the VM, eg:

* Set "Auto start" on dom0 boot

* Set memory limits, etc

h3. Start the domU

Use "Virtual Machine Manager" or the command:

<pre>

virsh -c xen+ssh://root@dream.csr.av/ start ${vm_name}

</pre>

h3. To do in a libvirt shell

Start a libvirt shell, with "Virtual Machine Manager" or with the command:

<pre>

virsh -c xen+ssh://root@dream.csr.av/ console ${vm_name}

</pre>

Log in as root in the libvirt console.

h4. Network config

Add @/etc/systemd/network/50-wired.network@ (adjust the IP):

<pre>

[Match]

Name=eth0

[Network]

Address=192.168.0.14/24

Gateway=192.168.0.10

DNS=192.168.0.10

</pre>

Then:

<pre>

systemctl enable --now systemd-networkd.socket

systemctl enable --now systemd-networkd.service

systemctl disable --now networking.service

</pre>

h4. Post-install ssh config

Allow ssh root login with password in @/etc/ssh/sshd_config@:

<pre>

sed -i -e 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config

systemctl restart ssh.service

</pre>

From this point onwards, one can close the console session opened via @libvirt@.

h3. DNS

Log in to the local DNS server with:

<pre>

ssh root@infra.csr.av

</pre>

Update the 2 files in @/etc/bind/zones@ (@zones/db.csr.av@ @zones/db.192.168.0@) and reload the DNS with:

<pre>

rndc reload

</pre>

h3. DNS

Log in to the local DNS server, *update the 2 files in @/etc/bind9/zones@*, and run:

<pre>

rndc reload

</pre>

h3. Allow yourself to log in easily with your ssh key

Copy your ssh key to the domU: run from your own computer:

<pre>

ssh-copy-id root@${ip_addr}

</pre>

h2. Ansible

Using the Ansible project developed in Blue Light: https://redmine.bluelightav.org/projects/ansible

h3. Prepare the host

Install Python

<pre>

apt install -y python

</pre>

h2. Database

The Postgis database runs on its specific domU (gisdb.csr.av, 192.168.0.18).

h3. Installation

After installing the Postgis package (eg. assign the "postgis" Ansible's role), follow up to [[Db-support]]

h2. Jupyter

The Jupyter server runs on its specific domU (jupyter.csr.av, 192.168.0.19).

h3. Installation

See #6990 .

h1. Backup and restoration of the database

h1. Database

h2. Troubleshooting

h3. Layers missing in the map's tree

Gisaf relies on counting features through Postgres statistics collector subsystem.

In case the server is restarted *dirty* (eg. without clean shutdown), then the count of the tables might be wrong or just 0, leaving the layers apparently empty and thus not even appearing.

The fix is as easy as:

<pre>

sudo -u postgres psql avgis -c VACUUM

</pre>

h2. Installation

This documentation assumes that the Postgis package has been installed (see [[CSR_server#Database]]).

h1. Configure the server

h4. Allow connections from other hosts in the local network

Set the server to listen to addresses, set listen_addresses to @*@ in @/etc/postgresql/9.6/main/postgresql.conf@.

Allow the connections, add in @/etc/postgresql/9.6/main/pg_hba.conf@:

<pre>

host all all 192.168.0.0/24 md5

</pre>

h2. Creation of the database

As @postgres@ user:

<pre>

createdb -E utf8 -T template0 avgis

</pre>

h2. Backups

h3. Primary

The database is backed up every day at midnight. The dump file is located in @/var/backups/postgres/@.

h3. Secondary

There are other backups (daily, weekly, monthly) thanks to Debian package @autopostgresqlbackup@), located (default) in @/var/lib/autopostgresqlbackup@.

h3. Tertiary (dom0)

The whole virtual machine is backed up by BackupNinja on the "dom0" controller, using:

- rdiff backups every day

- tar files on Saturdays.

See @/etc/backups.d@ on the dom0 (192.168.0.12).

h1. Remote

*%{color:RED} TODO: remote backup to be written%*

h1. Restoration

If the VM is not shutdown properly, there's a chance that the database is corrupt, and needs to be restored from one of the backups.

h2. After the restoration, restart gisaf:

<pre>

systemctl restart uwsgi.service

</pre>

h1. From primary backup

*Note*: the roles aren't restored with this method.

With user @postgres@:

<pre>

# Optionally, rename the corrupt database (selecting a name for a database like "avgis_c2")...

psql -c "ALTER DATABASE avgis RENAME TO avgis_c2;"

# ... or drop the existing database

psql -c "drop database avgis;"

# Create a new database:

createdb -E utf8 -T template0 avgis

# Restore the database

pg_restore -d avgis /var/backups/postgres/avgis.pg_dump

</pre>

h3. From secondary backup

@autopostgresqlbackup@ backs up the roles in @postgres_globals@.

<pre>

zcat /var/lib/autopostgresqlbackup/daily/postgres_globals/postgres_globals_2018-10-24_06h25m.Wednesday.sql.gz | psql

zcat /var/lib/autopostgresqlbackup/daily/avgis/avgis_2018-10-24_06h25m.Wednesday.sql.gz | psql

</pre>

h2. Gear

h3. Survey equipment

h2. See [[survey equipment]] - *%{color:RED} to be written%*

h1. Weather station

h1. Ambient Weather weather station

We have purchased a WS2902A weather station (https://www.ambientweather.com/amws2902.html).

Firmware version: 4.0.2.

h2. Manual

The operating manual of the weather station can be found at https://p10.secure.hostingprod.com/@site.ambientweatherstore.com/ssl/Manuals/WS-2902C.pdf

h2. Connection

h3. Wifi

Set up the wifi of the console using the "Ambient Tool" phone application. IP address given by DHCP on the router is: 192.168.1.101

h3. Local communication

Fail so far: the only exposed port is TCP/45000. Telnet doesn't show any activity. Nothing found on Internet on this protocol.

One interesting project may come, hijacking the connection to cloud services: https://www.wxforum.net/index.php?topic=35033.0

h3. Cloud connection

We'll create an account on AmbientWeather.net (and eventually on WUnderground.net and/or weathercloud.net), and:

* have the console upload data to there

* gisaf to retrieve our WS data from there

h1. Notes on Console

The daily rainfall data displayed in  the console resets at 00.30 every night

h2. Plan for future

Beside living well, 

h1. Plan

Some interesting projects that might be integrated:

* https://github.com/Oslandia/albion : Build 3D geological model from wells information

h2. Other

h1. GDAL (OGR) tools

"GDAL":https://gdal.org/ is a translator library for raster and vector geospatial data formats. It is used by many software (including QGIS and many other open source ones, including Gisaf). Some command line utilities are supplied, like:

* @ogr2ogr@ can easily convert one data format to another

* @ogrinfo@ displays information about files.

h2. Using Windows

On a computer with Windows and GQIS installed:

1. Open a command line console (eg. <Windows Key> to display the Start menu, then just type @cmd@ and <Enter>)

2. In the console window, type (adjust with the QGIS version and location, this seems to be the standard one):

<pre>

"c:\Program Files\QGis 3.10\OSGeo4W.bat"

</pre>

3. GDAL utilities can be used: @ogr2ogr@, etc.

h2. Example: convert Geopackage to Shapefiles

Output the content of the geopackage @9wdoogfr_2019-11-13_12_26_07.gpkg@ to the folder @shapefiles@:

<pre>

ogr2ogr -progress -f "ESRI Shapefile" shapefiles 9wdoogfr_2019-11-13_12_26_07.gpkg

</pre>

To output the content of the geopackage @9wdoogfr_2019-11-13_12_26_07.gpkg@ to the @root@ folder:

<pre>

ogr2ogr -progress -f "ESRI Shapefile" c:\shapefiles 9wdoogfr_2019-11-13_12_26_07.gpkg

</pre>

h2. With reprojection

Same as above, reprojecting to UTM44N: 

<pre>

ogr2ogr -progress -f "ESRI Shapefile" -t_srs EPSG:32644 c:\shapefiles 9wdoogfr_2019-11-13_12_26_07.gpkg

</pre>

h2. Links and references

h1. Links

h2. Water management

h3. Modflow

* https://water.usgs.gov/ogw/modflow/

The reference software for underground water modelling and simulation.

In conjunction with flopy (https://water.usgs.gov/ogw/flopy/) and Jupyter (https://jupyter.org/), it provides a _relatively easy_ to use interface.

h3. Freewat

* http://www.freewat.eu/project

This project is partly based on modflow, and integrates with QGis.

h2. QGIS

* https://www.qgis.org/en/site/

* Tools for Geology

Construction of geological cross sections in QGIS - http://www.geokincern.com/?p=1452

h2. Autocad

* Overview of Converting Geospatial Data to Drawing Objects: 

http://docs.autodesk.com/CIV3D/2013/ENU/index.html?url=filesMAPC3D/GUID-C38FD485-3CC2-4B52-8264-0D8C0F45422B.htm,topicNumber=MAPC3Dd30e41809

* CAD-DB connection:

https://knowledge.autodesk.com/support/autocad-civil-3d/learn-explore/caas/video/youtube/watch-v-AQoB--nyUJA.html

h1. Orfeo

* https://www.orfeo-toolbox.org/

Remote sensing

h2. Old docs

[[Shapefiles]] - *%{color:RED} to be written%*

h1. Data (measurements auxiliary tables)

Besides the importation of [[shapefiles]], Gisaf can import non-geophical information: auxiliary data (typically categories like the list of locations names, well types, etc), and temporal informations (well levels, etc).

h2. Command line

The @import_to_db.py@ script imports files, fetched from a set of URLs (typically, in the Redmine Files section of this project), formats and pre-process, and imports to the database.

@import_to_db.py@ is a support tool, that is planned to be integrated with the web interface.

Import all with:

<pre>

phil@phil-mbp:~/BlueLight/gisaf_src/gisaf$ python import_to_db.py

</pre>

The script currently accepts an argument for filtering the URLs to import.

h1. Pavneet's docs (imported from gisaf's wiki)

h1. Basic Rules of Map making

Regardless of the cartographic style or content, most maps have the following common elements.

h2. TITLE

The title should be in a large font, easily identifiable as the title of the map and should include descriptive text as to the location and purpose of the map. If the map is thematic, the theme should be included in the title. For example: Corn Production in Washington, 1990. The title is usually the largest font size of all lettering on the layout, however, it should not dominate the map graphic itself. The title may or may not be in a box and does not need to be at the top of the page (though it often is). For published materials (e.g., books or articles) the title may be included in a figure caption instead.

h2. SCALE INDICATOR

The scale of the map is typically indicated by a graphic bar scale, a representative fraction or a verbal scale. The reader must be able to determine the relationship between a unit of measure on the map and a unit of measure in the real world.

h2. ORIENTATION

A map should indicate which way is north (and/or south, east and west). Commonly this is done by a north arrow or compass rose. Orientation may also be shown by graticule or grid marks (e.g. lines of latitude and longitude). By convention north is towards the top of the page (thus some maps do not have north arrows), but the orientation must still be given for a 'proper' map. North does not have to be at the top of the page and a north arrow is essential in maps where it is not.

h2. BORDER(s)

A border identifies exactly where the mapped area stops. The border is often the thickest line on the map and should be close to the edges of the mapped area. The distance between the map and the border should be the same on all sides (balanced).

There can also be a border around the entire map layout (enclosing and grouping the title, legend, text boxes, etc.). 

Both of these borders are sometimes referred to as a 'neatline.' In addition, there is sometimes a thin additional line just outside of a border (accentuating it and ideally making it more visually appealing) that may also be referred to as a neatline.

h2. LEGEND

A legend defines the symbols or colors (including shades of gray and patterns) used on the map. Maps do not need legends if the symbology is so common or simple as to be easily understood by the reader. However, it must be clear what each marker or line type, weight and pattern represents. The legend does not need to be labeled "Legend." The more complicated the symbology on a map the more important the legend becomes.

h2. MAP CREDITS

* SOURCE OF DATA (especially on thematic maps)

* NAME of the cartographer  

* DATE of the map creation/publication 

* DATE of the map data 

* PROJECTION of the map (especially small-scale maps)

h2. LOCATOR MAP (INSET)

A locator map is needed if the area of the map is not easily recognizable or is of large scale. For example, if you map Whatcom County, there should be an inset map of Washington, showing the location of Whatcom County.  Inset DETAIL map(s) may also be used to show an area of the map in greater detail (larger scale).

h2. EFFECTIVE GRAPHICAL DESIGN

The layout design is as important as effective sentence structure is to written text. Layout design refers to the planning and decision making processes involved in the visual display of the spatial data. You can achieve balance by rearranging the map elements (north arrow, legend, scale, title, etc.) and changing size of the text, border. etc. The map and map elements should be:

* Neatly drawn

* Appropriately and consistently generalized

* Symmetrically balanced (avoid crowding or large blank areas)

* Without unnecessary clutter (keep it simple, be wary of 'artistic' details)

h2. VISUAL HIERARCHY

A hierarchy of symbology should be used for the lettering, line weights and shading. More important features are typically larger and/or darker, less important/background information should be smaller and/or lighter. At the same time, do not "over weight" or "under weight" features.

h2. PURPOSE

All maps have a purpose which should influence every element of the map and the map layout. A cartographer should be able to clearly articulate the purpose of their map and should keep the audience (who the map is going to be used by) and the client (who the maps is being produced for) in mind.

h3. NOTE: Any, or all, of the above 'rules' can be (and frequently have been) violated at the discretion of the cartographer IF doing so produces a better map (better serving its purpose and audience).

*In general, with cartography, less is more (avoid excessive clutter).*

h1. GIS - Survey_Database

DB- Database

C3D- Civil 3D

h1. How to create Survey database in Civil 3D

h2. 1. Setting up the Working Folder

A working folder needs to be created where the survey databases gets stored. In the *Toolbox* > Right click on *Survey databases* > Set the working folder - Save the folder in the desired location. We are setting the Working Folder for the Civil 3D databases in C:

!https://redmine.auroville.org.in/attachments/download/9783/working%20folder%20image.jpg!

h2. 2. Creating a database

Right click on *Survey databases* > *New local Survey database* - Enter the name> click ok. A survey DB is created (It is in bold which signifies it is the current DB under use). Multiple DB's can be created. A survey DB can be opened for editing by right click > *open for edit*. It can be closed by right click > *close for edits*. A survey DB can also be opened as *read-only* by same procedures.

h2. 3. Setting up the database

Survey Database(abc) > Right click + Survey Database Settings > Specify co-ordinate Zone > Distance- Metre > Temperature - Celsius > Pressure - Millibars > Distance type - Horizontal > Vertical type - Vertical Distance.

Then the next step is to create the Networks under which the point data gets uploaded. For example in our case we have - TS and RTK

h2. Components of Survey Database

Survey database in civil 3D has the following components when expanded. 

# Import Events

# Survey Queries

# Networks

# Network Groups

# Figures

# Figure Groups

# Survey Points

# Survey point Groups

!https://redmine.auroville.org.in/attachments/download/9782/Survey_db.png!

h1. Survey- Field to Finish

h2. Steps from field work 

# ?? To be added by Raj and Ram

# Copy the .txt files from the equipment- Controller/Total station, to system using a pendrive (Storing the data- "D:"> "AVSM"> "Water Projects"> Respective project folder)> "TS"/"RTK")

# Cleaning up the .txt files/removing errors (If any)

h2. Feeding the survey data into Civil 3D- by Surveyor

# After cleaning up of files, the files are stored in a desired location, to be used as import events into the Civil 3D file

# Open up project in Civil 3D

# Importing of Events (explain the step)

h2. Processing- by Surveyor

Initial line work is generated from survey points in the drawing. Points for reconciliation (Changing the point codes) are reported, if any. It is important that there is a fair knowledge of Autodesk Civil 3D for this work.

h2. Post-Processing

Final linework is generated on top of the processed linework. At this stage, a drawing is cleaned and prepared for sharing. problems like - overlaps in line-work, proper assigning of layers etc. Points for reconciliation are reported, if any. Post-Processing is crucial to generate correct and standardised survey drawings (in .dwg format) keeping in mind that those drawings will be used to generate shape files (.shp format). Therefore, it becomes important to follow a certain workflow based on compatibility of .dwg elements and .shp elements - for example for creating a shapefile of "Polygon" type, the elements in .dwg should be all "Polylines" of "closed" nature. All features should be in zero " 0 " elevation and the shapefile generated should be 3d type. This has been elaborated here < insert > At this stage, a good knowledge of Autodesk Civil 3D and GIS (to a certain extent) is a must.

h2. Creating shapefiles in AutoCAD

?? add by pavneet - *%{color:RED} to be written%*

h2. Sharing on WebGIS - "GISAF-  +gis.auroville.org.in+" 

After creating the shp files from AutoCAD, the shp files are ready to be shared on WebGIS platform. Following are the steps to do so.

# *Upload and import Raw survey points to GISAF* - Before uploading and importing the shapefiles, it is necessary to upload and import the corresponding Raw survey points into GISAF.

# *Auto-import of point shapefiles* - After the import of point files (RTK and TS), the next step is to Auto-import the points which are point type shp files. Go to Admin> Others> Projects> Select the corresponding project > With selected > Auto- import to GIS Database ( GISAF)

# *Upload shp to basket* - Zip together all the types of files obtained after saving a shapefile on the system (shp, prj, dbf etc). Follow the same naming standards when creating this. Upload it to the basket in GISAF- Admin> Basket> Shapefiles> click on relevant project> click on relevant sub-project (if any)> upload file> select the zipped file from system> select the category from directory >save.

# *Import to GISAF (Only with Authorization)* - Click on the import arrow.

h2. Adding new codes

ADD !! - *%{color:RED} to be written%*

h2. Reconciliation of points

The points are noted with their codes from and codes to reconcile into. ADD!! - *%{color:RED} to be written%*

h1. Survey Data Post-Processing

+Softwares used: Civil 3D, QGIS+

AutoCAD Civil 3D - C3D, 

C3D is being used for post processing of survey data. C3D offers a BIM solution for Land Surveyors. Basic knowledge about C3D can be obtained by following tutorials online though a pre-acquired knowledge of Autodesk AutoCAD is a plus point and sufficient enough to start working with C3D.

QGIS is an Open source GIS.

Getting Started:

+Standards and Workflow for C3D+

The layers (Nomenclature of layers is coming from standards- U.S. National CAD Standard Version-AIA) are pre-assigned a geometry type (Point, Line and Polygon) in their layers description (Use layer manager to check). This is done keeping in mind inter-operability with GIS (In shapefiles format), geometry type Shape files are of three kinds- points, lines and polygons.

+Geometry types-+

Based on the description of Layers, if the layer Geometry is specified as 'Point', the raw survey data can be auto imported to webGIS - *GISAF*, no post processing is needed for 'Points'. For lines and polygons, We use Polylines (for 2D lines/Polygons) and 3Dpolylines (for 3D lines/polygons). In case of curved lines in 2D, Polylines command (PLINE/PL) is enough but its not possible to create curved lines in 3D using the 3Dpolylines, in such cases feature lines come into the picture. (*Important:* 2D polylines, splines, ellipses and circles are not exportable to shapefile format, they have to be always converted into Polylines).

*We are at the moment generating shapefiles (lines and polygons) in 2D (z=0) but keeping the format of files as 3D.

+Using Feature lines for creating Curved geometries in 3D-+

There are a series of steps involved when creating curved lines using feature lines. You can follow two methods depending on the situation :-

Method 1(For curved objects in 3D using elevation of points):- Create the object using 3DPOLY. Now use the create feature lines from objects and select the objects to be converted to feature lines. You will not notice any difference in the geometry on the screen as such but in properties you will see the object type description of selected object as feature lines.

Method 2(For curved objects in 3D by draping the lines/curves on a surface):- Create the object using PLINE. You will notice all the lines are straight. Now use the create feature lines from objects and select the objects to be converted to feature lines. You will not notice any difference in the geometry on the screen as such but in properties you will see the object type description of selected object as feature lines.

Now modify tab is used to convert the straight lines to curved lines. There are many ways to do it and different methods can be adopted in different situations. The easiest and the most commonly used method is by using the smooth command in the modify tab. After using the smooth command, a curve would be visible. This is a curve in 3D. To view it, object viewer option can be used by selecting the object and right clicking.

More complex methods have to be studied and each case has to be taken into consideration separatly when using - FITCURVEFEATURE In some cases to obtain a desired curve(Meaning more fragments) more PI's (Point of insertion) need to be added into the feature line. PI are the points where the feature lines gets fragmented when exploded.This step often becomes difficult to manage due to complexities involved in mathematical functions behind creation of feature lines.But it should be explored by all means.

Because the feature lines are not exportable to shapefiles, the feature line has to converted back into a 3D polyline. This can be done simply by explolding the object. Once exploded into 3D polylines, you will see that the curve gets fragmented into smaller segments of straight lines. You can repeat the process of creating the 3Dpolyline object into feature line and then smoothening it till you get a desired geometry resembling the curve.

Situations of sharing of points with different layers and proceeding with linework in such circumstances-

There is always a best way to take the survey points taken in the field by the surveyor. For instance, in case of a road and a curb adjacent to it, the surveyor takes the points only once. It is only during the post processing that the lines are generated (You can create the lines in field while taking the survey points as well, but that is not the most feasible method majorly due to time constraints) and in this case, two line on the same position shall be generated, one under layer of roads and one under layer of curbs. 

The line work becomes complex in case of generating 3d lines and 2d lines on the same place. This can arise due to having some layers as 2D (for example building outlines) and some as 3D (paved surfaces- <taking into consideration the future use of the layers. In this case, for the purpose of water management, it becomes essential to have survey of such surfaces as 3D. Survey of building outlines doesn't need to be 3D because the elevation points are not recorded while surveying them due to feasibility).

So, in case of generating 3D linework from survey points with no elevation, Feature lines and SURFACES come into picture. The surfaces are generated by an interpolation method in C3D by giving a set of Point Groups. We have selected the triangulation method as the appropriate method due to availability of a dense set of points. 

While creating the feature lines from objects (as explained above, check the box "assign elevation" and a dialog box appears to select the surface you want to select). Surfaces will have to be generated prior to using this. (add wiki for generating surface). Surface should be generated including the points of the layer in the point group of it. 

*+Purpose of Survey+*

# The survey is conducted for generating a Base Map which is representative of Topography, Infrastructure, Drainage, Buildings etc, with a focus on Water Management in Auroville. Furthermore this survey can be used as base map to build upon more detailed surveys for the various purposes such as Town planning, Land Surveys, Transportation Planning etc.

+Use of Layers+

# *V-BLDG-SHED* (Shed) layer is used when the structure is not entirely enclosed, if At least one side is open (No walls/partition etc). One thumb rule is to answer the question if the building can be locked safely or not. If not, then it is a shed.

# *V-BLDG-HUT* (Hut) is used when the building (For human activity/living space) is single storey with roof made up of perishable material such as keet, straw etc.

# Use of *V-BLDG-ROOF* (Roof) is for buildings with permanent roofs extending out about more than 1 meter from the outline at ground level. It can also be used for complex roof structures for example Matrimandir Petals.

# Use of *V-BLDG-OTLN* (Building outline) is for depicting the outer line of walls of a structure on ground level. This is used when the roof of the building is more or less of the same profile.

# *V-BLDG-RTWL* (Retaining Wall) is for walls with the function of retaining either earth/water and the side of the wall should be visible. In case of very thin walls and no side visible, it comes as *V-BLDG-RWLL* (Retaining wall line). This is depicted as a single line in drawing.

# Use of *V-ROAD-CYCP* is for designated cycle path as a single line.

# Use of *V-ROAD-FPAT* is for pedestrian paths as a single line. It is used when there is a path of very small width. If the need is to represent a path/road which is unpaved in nature, *V-ROAD-UPVD* is used, which is polygon type. To represent a path/road of paved nature *V-ROAD-PAVD* is used, which is also a polygon type.

# *V-WMNG-PIPP* for pipe points, *V-WMNG-PIPL*- for pipe lines.

# *V-BLDG-BMRK* is for unidentified bench marks and *V-BLDG-MHOL* is for unidentified manholes.

# *V-BLDG-PLTF* is for impervious surfaces. for example sand and gravel will not come under this layer. cemented paths etc can come.

# *V-WMNG-POOL* is for water bodies made for leisure activities- like swimming pool etc.

# *V-WMNG-SUMP* is collectively for artificial water retention structures.

# *V-WATR-POND* is for top of natural water retention.

# *V-WATR-PONB* is for bottom of natural water retention.

# *V-WMNG-DRBA* is for bottom of artificial drains.

# *V-WMNG-DRTA* is for top of artificial drains.

# *V-WATR-DRBN* is for bottom of Natural drains.

# *V-WATR-DRTN* is for top of Natural drains.

# *V-WMNG-DRNC* is for drain covers.

# *V-WATR-DRAS* is for indicating slope of drainage, these are arrows.

h1. Wells Documentation

Documentation of the wells in Auroville started afresh in September 2017 by Bala working with the *Water Group*. He used a mobile GPS to record co-ordinates and took pictures along with other data related to a well. An effort was made to reconcile data by identification  of the wells with existing data from *Auroville Water Harvest* which ceased to exist around 2007/2009. In some cases the codes on the well on location (sometimes there is a code on the pump, on the casing or on a nearby wall) helped in reconciliation but mostly by spatial mapping using QGIS. Some wells could not be reconciled due to absence of any nearby well in old data. However, we are publishing all the wells we have documented so far.

*Terms of reference*

+Well status+ 

# *In use*      Well is equipped with functioning a pump	

# *Not in use*  Well is not equipped with a functioning pump. (Special case- well is equipped with a non functioning pump)

# *Closed*      Well is closed/ sealed/ abandoned, (Historical reference)

Note: Wells with pump under repair (temporary measure) at the time of survey are treated as *In use*.

h1. Civil 3D useful commands

h2. Making Feature line to polyline-

Select the feature line> go to elevation editor (under feature line tab> edit elevations)> select all the points in table, give elevation(this will give same elevation to all the points thereby making it possible to retain the curves in the polyline) > explode the feature line (use X enter)

h2. Converting Circle to polyline-

BR > enter > break the circle at two points to obtain a part of circle. 

PEDIT> enter> select the part of leftover circle> J > enter> Close> enter.

h2. Converting 2DPOLYLINE to POLYLINE

Explode the 2D polyline, and use the PEDIT command to convert the segments to polylines. Then join the polylines.

h2. Viewing only the used layer in ACAD-

Set the value of SHOWLAYERUSAGE from 0 to 1

h2. Convert 3d polyline to polyline- 

COVERT3DPOLYS (Change the elevation of the polyline using properties manager to 0 in our case)

h2. Generating contours from a Surface in Civil 3D

# In prospector, Create a point group of points needed to generate the contours from for example- TOPO elevation points or BLDG floor levels. If the point group is already present, then proceed to next step.

# Create a surface in the prospector> when creating the new surface, give the name and styling in the dialog box- styling used by us is "Triangles and surface- 0.1 and 0.5"> Assign the point groups in the surface as created in previous step.

# Go to the surface created in prospector and right click > Edit surface style> Turn on the Major and Minor contour.

# Turn on the layers for Major and Minor contours in Layers Manager (LA > enter)

h2. Shortcuts:

• Properties manager - ctrl + 1

• To view 3D- Select and right click> object Viewer.

• To copy the Line - Copy and select the line enter 

• To convert spline to polyline >Splinedit

• To export shapefile - Mapexport > follow the process.

• To assign coordinate system to the drawing - MAPCSLIBRARY

• To check coordinate system of the drawing- > TOOLSPACE> Settings> Right click on the drawing name > Edit settings

h1. Online references for Civil 3D

*Description Key Sets*

https://www.youtube.com/watch?v=Von8oCwYcTk

*Autodesk civil 3D Geotechnical module- for borehole data*

https://www.youtube.com/watch?v=0Be9kShBou0

*Civil 3D for Surveyors*

https://www.youtube.com/playlist?list=PL1EzH8XlwSxuxCMdeLvzqfgsAlmeeHMv2

*Civil 3D: Survey - Survey Database*

https://www.youtube.com/watch?v=VPWdAfYJt5Y 

h1. Connections in QGIS- Using browser panel and Add postGIS

Working using QGIS as interface using PostGIS connections.

* Server: @gisdb.csr.av@

* Database: @avgis@

* Leave @Service@ empty

h2. Adding tables (With geometries) in form of shape files from database using Browser panel-

Go to QGIS> view> Panels> Browser panel> PostGIS> Expand the connection> give credentials> add the desired file by double clicking on it.

h2. Adding tables (For non-geometry type) using PostGIS connections-

Add PostGIS Layers > Give credentials > select "Also list tables with no geometry" > expand public > click on the desired table> add.

h2. Joining tables

Right click/double click on file> go to Join > perform the desired joins - add/subtract the joins. 

h1. *Reconcilation of Raw survey data using pgAdmin*

+For changing the layer codes, to be done using Pgadmin.+

h3.  Enter Schema> AVSM RAW Survey > Tables > select the table> Right click - View edit data > All rows. Apply filter to the original id and note the Database id's to be reconciled.

h3.  GISAf Admin> Other > Reconciliation. Create > Add the database (point) id and give the new target corresponding to the database id to be changed. (Use the Layer name in target not Raw layer name).

h3.  Other > Project > select the project > with selected> reconcile RAW survey points.

h1. Importing point data (TS and RTK) to GISAF 

# *Gisaf Admin > Basket > Survey data > "Project" > "surveyor" > TS/ RTK > upload > import.*

# *Auto import of Raw points data* (changes from Raw to Shapefiles, the point files)

  In GISAF Admin > Other> Project > select the project for which you want to import the data > with selected > Auto import to GIS...

h1. Editing Z value of features in Shapefiles in QGIS

h2. using vertex editor tool - https://www.youtube.com/watch?v=8V8i1AtcA74&t=256s

h1. Miscellaneous- Civil 3D

*Autodesk civil 3D Geotechnical module- for borehole data :- To analyse borehole data, To make profiles and calculate volumes*

https://www.youtube.com/watch?v=0Be9kShBou0

Reference videos

Exploring:

https://www.youtube.com/watch?v=gr-ISPzLcU0

Modelling:

https://www.youtube.com/watch?v=Pz0-HOoiBrs

solids

https://www.youtube.com/watch?v=mQ0Yeh6tZA8

https://www.youtube.com/watch?v=4eMsUiYBhuE

*CIVIL 3D Survey*

Getting started- Always open a new drawing with a template. 

*Description key sets*- (till 18:30) https://www.youtube.com/watch?v=mmwkkRyBkS0

As when the points come into the drawing, they are going to be filtered with description key sets and the description key sets will assign properties to the points.

Tool space > settings > points > description key sets

“Description key sets name”> edit key- to view the points in a list: 

Automatic linework (18:30- till end)

Survey > Linework code sets- for automatic linework

*Using Master view*

Uses-

copy styles from one drawing to other

copy drawing data from one drawing to other (data referencing)

*Civil 3d surface model*- from points- break lines

https://www.youtube.com/watch?v=wC40rdmDEMo

moving for example- point groups under point groups- 

*Civil 3D Planning and Analysis*

Workspace: Planning and Analysis

1. Working with Object data (GIS Attributes)

Map Setup> Define object data> New table> Define new object data table start defining fields

h1. Documentation- Rain Gauge

h2. Manual Rain Gauge

*Why it is important to comply to standards?*

1. Consistency for comparing rainfall data in different places within Auroville since rainfall varies in different parts of Auroville.