ETL process
ETL (Extract, Transform and Load) is a process in data warehousing responsible for pulling data out of the source systems and placing it into a data warehouse. ETL involves the following tasks:
- extracting the data from source systems (SAP, ERP, other oprational systems), data from different source systems is converted into one consolidated data warehouse format which is ready for transformation processing.
# - transforming the data may involve the following tasks: applying business rules (so-called derivations, e.g., calculating new measures and dimensions),
# cleaning (e.g., mapping NULL to 0 or "Male" to "M" and "Female" to "F" etc.),
# filtering (e.g., selecting only certain columns to load),
# splitting a column into multiple columns and vice versa,
# joining together data from multiple sources (e.g., lookup, merge),
# transposing rows and columns,
# applying any kind of simple or complex data validation (e.g., if the first 3 columns in a row are empty then reject the row from processing)
- loading the data into a data warehouse or data repository other reporting applications
