F21EP - Data Engineering Pipelines

At the end of the course, students will have experienced an end-to-end data processing technological stack, from collection to reporting.

1. Data Sources: Primary (sensors, surveys), Secondary (data formats, APIs), Quantitative, Qualitative

2. Data Storage: RDBMS, NoSQL, Data Warehousing

3. Data Processing: Data Indexes (forward, inverted, spatial), Data transformation and integration (ETL, ELT, ETLT)

4. Spatial Foundations: Vector / Raster data, spatial predicates, spatial functions, PostgreSQL + PostGIS, map visualisation (GIS)

By the end of the course, students should be able to do the following:

establish the capabilities and limitations of data sources, data models and database management systems.
demonstrate competencies around the use of software tools and languages in the design, development and management of data processing pipelines.
appraise data integration and processing applications critically to compare different systems and establish their capabilities.
construct spatial-temporal databases and demonstrate competency in spatial-temporal data analysis and visualisation.

Further details

Curriculum explorer: Click here

SCQF Level: 11

Credits: 15