Seamless transition from business analyst to developer
Discovery Studio is a complete suite designed for data analysts within data platform development teams to discover, profile and select the data sets that are necessary to build a datawarehouse solution. As part of making data stories ready for delivery the Discovery Studio creates a seamless transition between the business analysts and the data developers in your data platform teams.
Add data sources
Easily add your data sources via the data connection wizard. We support API, database and file based connections. SEPIDATA also provide specific API’s to connect to specific governments and major software vendors.
Profiles data sets
Gain understanding of the content of a data set, run profiling to get insights on the data. Get advice on the data structure when these are not supplied
Apply business terminology
Add data transformations
Add calculated fields
Send to data developers
Introducing the Discovery Studio
The power of Visual Studio
The SEPIDATA developer studio contains project types, item templates, msbuild integration and developer tools to build a datawarehouse solution for the SEPIDATA datawarehouse automation platform. It also includes an integrated runtime engine for rapid iterative development and testing cycles within the Visual Studio environment. Equipped with all the great features of Visual Studio developing and deploying your data warehouse was never so easy and fast.
- Support for Visual Studio 2017.
- Support for SQL Server 2016 and higher.
- Define your datawarehouse objects and structures in JSON.
- Build you application and get warning and validation errors as if you are coding a .NET application.
- Deploy your solution and run and test it directly after you make changes.
- The visual studio embedded intellisense helps you in picking the correct options for settings on specific DWH objects.
- Build your DWH solutions by creating projects that reference each-other like normal application development.
- Get errors and warnings when you build your DWH solution in Visual Studio.
- The JSON file is validated: does it contain all options and properties for the given object and is it consistent with the rest of the model(s).
- We embedded the rules of the DWH methodologies in our custom build compiler so when you build the solution we protect you from common pitfalls
Build a DWH solution
- Source Model: With this project type you can describe the source system or data model. You can add tables, views, stored procedures and files that are present at your source and where we will retrieve the data from to load it in the staging area.
- Staging Model: With this project type you can define an interface and a target staging area which uses the above described source model. Within the interface you map the structures from your source model to the staging area. Within the staging area you can configure all staging settings: for example if you want to historize the specific table, what type of mutations you would like to detect and much more.
Manage the work of the team
Because our developer studio is build on top of Visual Studio we provide your datawarehouse development team with all the great features of Azure DevOps (on premise or in the cloud). This way you are equipped with a lot of powerfull features for managing the work of your team.
The backlog management capabilities integrate perfectly in Visual Studio and allow you to assign work items so you can relate datawarehouse changes to the backlog items for a perfect audit trail of all changes. This way not only the “how” is visible, but the team and auditors can also see years after a change what the reason was behind a certain change in the datawarehouse.
Meta code your datawarehouse
We use JSON combined with smart wizards to develop your data warehouse solution. JSON is easy for humans to read and write and it is easy for SEPIDATA to parse and generate large pieces of DWH code for you. Within the SEPIDATA developer studio datawarehouse objects like a source table or a staging object are coded or better said “described” in a JSON based format. Every object has its own (by SEPIDATA defined) outline in JSON.
At this moment we do not have any project types that require data transformation because the staging area stores RAW source data. To let you know upfront, when the next project types arrive that do need data transformation (for example the Data Hub or Datamart), you can transform your data by writing SQL code. This SQL script will be embedded within a JSON described transformation object. Around these objects we will provide functionality to parse the script where we will catch the output structure of the query and describe the output structure in a JSON format for you, we will even grab the lineage out of the scripts and add this to the JSON described object. This way you can focus on what’s important: understanding your customers data needs!
True developer guidance
Within our developer studio we provide the usual features around coding and compiling. You can build a specific project or the whole solution. You will receive warnings, errors and informational messages that come from our solution compiler. This way you get feedback on the quality of the things that have been developed. We will keep on embedding the rules of the different DWH methodologies in our compiler so we can warn you about possible bad choices or mistakes that will fail your production DWH loads.
We also use the Project References. The staging area project type will always be based on a source model project. Our software will add the project reference so our compiler understands the build dependencies between projects.
We have intellisense implemented to help you choosing options for certain properties of DWH objects. Off course code completion and all the great shortcuts and coding features that are standard in visual studio can be used and will make the life of your developers a lot easier and more productive!
We have the best source control
We claim to have the best version control support of all DWH software / ETL tool providers. Your team is free to choose for GIT, TFS as repositories and we support Azure DevOps in the cloud but also the DevOps Server on premise.
When you choose for a TFS repository, you can use the exclusive locking (with Team Foundation based source control). Or if you are more advanced you can go for a GIT repository allowing you to code on a local repository syncing all the changes.
If you require you can also add check-in notes and policies to your project for mandatory documentation of changes. To enhance the quality of your development documentation and audit trail.
Please visit this page of Microsoft on comparing TFVC and GIT. Do you want to learn more about git please visit this site of Microsoft. If you want to know how Microsoft uses GIT please visit this site.