Stonemeadow Solutions ETL Framework

Abstract

This ETL Framework has been used by Stonemeadow Solutions clients since 2006 to minimize their ongoing Extract Transform and Load (ETL) Total Cost of Ownership. It supports SQL Server Integration Services environments and contains dynamic configurations, custom logging and a set of SSRS reports for monitoring completed and active SSIS activity.

This release is the foundational component of a larger SSIS solution focused on reducing SSIS development costs, this first version include the ETL Framework and an AdventureWorks sample scenario. The current version of the Stonemeadow Solutions ETL Framework includes:
  • A Zip file containing SSIS package templates, an AdventureWorks ETL scenario, an installation procedure that creates and initializes the ETL Framework database, a SSRS solution containing reports displaying ETL Framework custom logging
  • Users Guide - This document provides an overview of the Stonemeadow Solutions ETL Framework as well as a Developers and Operations Users Guide

The screenshot below, taken from the User’s Guide, is a Summary report. Note that this report supports navigating to more detailed information including: activity summary, errors, record counts and dynamic configuration values.

BatchHistoryClickThrough_Small.jpg

Following versions will provide tools to easily generate simple ETL (there will always be a need for custom SSIS development). This will include:
  • SSIS code generator – This uses an XML input to generate SSIS incremental load dataflows which are then inserted into ETL Framework SSIS package templates
  • User Interface – This allows users to easily generate the XML formats by specifying the source, destination and special columns, e.g. business (natural key), SCD columns.
These will be added to this project in the first quarter of 2011.

Many ETL implementations already have an ETL Framework in place. Please provide feedback indicating what you like and do not like about this ETL Framework. This feedback will be incorporated into future releases.

Last edited Dec 15, 2010 at 1:30 PM by Larry_Barnes, version 10