Create and design your own datawarehouse eco system
With the SEPIDATA datawarehouse platform you can manage large parts of your datawarehouse eco system. The focus of the SEPIDATA platform is on the creation and delivery of the analytical data model used to derive business intelligence. By delivering meta data API’s our system can integrate with data management solutions to exchange information about models, lineage, interfaces, source systems and more. Add a datawarehouse front-end (BI) on top of that and the eco system is complete.
Prepare for your datawarehouse
Get a clear overview of information requirements (target consumer types and their tooling, is there a need for capturing history, what regulations are applicable, what non-functionals are appplicable).
Setup a product backlog, the team & sprint configuration. Start creating epics and (data) stories and assign them to future sprints. Define the initial team velocity and prepare prioritation, estimation and sprint planning.
Architecture & governance
Define your datawarehouse solution architecture, data architecture, data governance and select DWH solution building block that best fit the requirements. Setup data governance within the organization.
Build and configure the environment(s)
Install database server(s) from development to production (depending on sizing and volumes a separate application server is recommended). Install the SEPIDATA runtime agent on an application or on the database server (depending on sizing).
Make sure that developer machines have Visual Studio and SQL Server developer edition installed. Download and install the SEPIDATA visual studio extension on developer machines.
Decide on the source control solution and the branching & merging strategy. Create an empty DWH solution in Visual Studio with our extension and add it to source control. Add the required DWH project types (based on your solution architecture) to the solution.
Incremental information delivery
Data discovery & staging
Discover data sources (database or flat files) by manually or via a discovery wizard. Setup the staging area, configure historical settings. Run data profiles on the RAW data. Everything is ready to ingest the source data into a new or existing staging area.
Design and build target model
Based on the methodology that is chosen design the target data model where data needs to be loaded to. Add the objects to the project via wizards or manually.
Build the transformation layer
Build the source to target mappings (transformations SQL statements). If applicable, do this for every applicable source that needs to be integrated into the target object (multi source strategy). Connect the target model (depending on methodology) to the transformation layer output.
Test and load
Build and compile your solution and resolve any compilation errors and warnings. Test and load the solution on your local developer machine, fix bugs or loading issues. Validate early prototype with data stewards / business users and correct based on feedback.
Publish, package and deploy
Publish the code changes to the main branch. Create a deploy package within Visual Studio or deploy via build server / pipeline. Deploy your solution towards acceptance environment. Run the datawarehouse load and validate the output with business / stakeholders (business acceptance test).
Publish the new DWH release to the SEPIDATA production application server and start building the BI front-end (reports / dashboards) or outgoing interfaces based on the new release of the information model.