Sifiso Ndlovu

Author
Recent Posts

Sifiso is Data Architect and Technical Lead at SELECT SIFISO – a technology consulting firm focusing on cloud migrations, data ingestion, DevOps, reporting and analytics. Sifiso has over 15 years of across private and public business sectors, helping businesses implement Microsoft, AWS and open-source technology solutions. He is the member of the Johannesburg SQL User Group and also hold a Master’s Degree in MCom IT Management from the University of Johannesburg.

Sifiso's LinkedIn profile

View all posts by Sifiso W. Ndlovu

Dynamic column mapping in SSIS: SqlBulkCopy class vs Data Flow

February 14, 2020 by Sifiso Ndlovu

The Data Flow Task is an essential component in SQL Server Integration Services (SSIS) as it provides SSIS ETL developers with an ability to conveniently extract data from various data sources; perform basic, fuzzy to advance data transformations; and migrate data into all kinds of data repository systems. Yet, with all its popularity and convenience, there are instances whereby the Data Flow Task is simply not good enough and recently, I got to experience such inefficiencies. To demonstrate some of the limitations of SSIS’s Data Flow Task, I have put together a random list of Premier League’s leading goal scorers for the 2019-2020 season.

Monitor batch statements of the Get Data feature in Power BI using SQL Server extended events

July 1, 2019 by Sifiso Ndlovu

One shared characteristic among popular data visualization tools such as Power BI is the ease at which data can be extracted from a variety of disparate data sources, usually at a click of a button, as shown in Figure 1. Such convenience, though, tends to come at a cost as you often have little control over how background scripts used to extract data are generated. Yet, this should be of utmost concern for data architects and BI developers alike as rarely do you find auto-generated scripts that are efficient and optimal. In this article, join me as I put on my DBA hat and trace, monitor and review SQL batch statements that are auto-generated by the Get Data feature in Power BI using SQL Server Extended Events.

Bulk-Model Migration in SQL Server Master Data Services

May 30, 2019 by Sifiso Ndlovu

Just over a year ago, I published How to migrate SQL Server 2017 Master Data Services Models into another server article, that essentially demonstrated the migration of a SQL Server Master Data Services (MDS) model from one MDS instance to another. Like many of my previously published articles, I have since used the aforementioned article as a technical reference (i.e. syntax lookup for using MDSModelDeploy.exe utility) during the implementation of MDS-related projects for my clients.

Using Power BI Free License to Embed Power BI into Dynamics 365

May 24, 2019 by Sifiso Ndlovu

Whilst Power BI continues to be the leading data visualisation tool in Gartner’s BI quadrant, Dynamics 365 for Customer Engagement (formerly Dynamics CRM) remains one of popular business platforms for driving sales and facilitating customer communications. Thus, an integration of Power BI into Dynamics 365 for Customer Engagement ensures that you get to explore your Dynamics 365 for Customer Engagement data via a powerful and rich data visualisation tool. However, as indicated in Figure 1, Dynamics 365 for Customer Engagement plans often come with a Power BI free license thereby limiting the type of Power BI features available to you. In this article, we look at how you can get the best out of your Power BI free license for the purposes of embedding Power BI artefacts into Dynamics 365 for Customer Engagement. Going forward, any references to Dynamics 365 for Customer Engagement plan in this article will be shortened to Dynamics 365.

An MDS Driven Approach to a Turnaround Time Calculation in SQL Server

November 30, 2018 by Sifiso Ndlovu

One calculation that you are almost guaranteed to have to produce in your career as a T-SQL developer relates to the calculation of a turnaround time. This is often a key KPI for measuring the performance of both individuals and teams, particularly when the business operates within a service-oriented sector i.e. customer support, transportation, healthcare etc. Turnaround time calculation does not only refer to business metrics rather any activity (i.e. ordering a pizza) with a recorded start and an end time can have its own turnaround time calculated. In this article we evaluate different options for calculating a turnaround time including using DATEDIFF function, creating your own user-defined function (UDF) as well as an integration with SQL Server Master Data Services.

How to Programmatically Pass Credentials in an Embedded Power BI Report

November 20, 2018 by Sifiso Ndlovu

In the article, How to embed a Power BI Report Server report into an ASP.Net web application, we looked at available options for embedding a Power BI Report Server report into an ASP.NET web application. Since the publication of the article, I have received several questions relating to how one goes about programmatically passing credentials for report server connection within an embedded Power BI Report Server report. The simple answer to such questions is that it is currently not possible to implement user impersonation in an embedded Power BI Report Server. However, like in most scenarios, there are workarounds that one could temporarily employ – at least until Microsoft comes up with a permanent solution to what is becoming a top requested feature at ideas.powerbi.com.

Replicating Excel’s XY Scatter Report Chart with Quadrants in SSRS

October 25, 2018 by Sifiso Ndlovu

In this article, I’ll share some of the tricks and tips that I had to employ in order to successfully replicate Excel’ XY Scatter Report Chart.

How to embed a Power BI Report Server report into an ASP.Net web application

September 20, 2018 by Sifiso Ndlovu

Every once in a while, teams from different functional areas of the business (i.e. business intelligence, software development, web development etc.) would join forces to form a cross-functional development team with a common goal of integrating a business intelligence artefact such as a SQL Server Reporting Services (SSRS) report into a front-end web application. There are many reasons for forming such a partnership including a lack of report-development skill by web developers, BI team owns a better reporting tool for data visualization, or maybe to prevent the software team from “reinventing the wheel” by developing a report that has already been produced elsewhere.

Available options for generating heatmaps in an SSRS report

September 17, 2018 by Sifiso Ndlovu

As developers of business intelligence solutions, we tend to service different types of business users. Whilst a call center supervisor is more likely to consume reports that offer detailed breakdown of day-to-day performance of her call center agents, senior managers on the other hand often prefer big-picture analytical reports that represent data in a form of colorful graphs and charts. Not surprisingly, such reports often make use of heatmap controls to provide visually effective comparative view of business metrics against set targets.

How to handle SSRS multi-value parameter filtering in SQL Server Parallel Data Warehouse

August 23, 2018 by Sifiso Ndlovu

Experienced business intelligence (BI) developers would tell you that as you move from one project to another, some requirements start becoming repetitive like you have dealt with them before. One such repetitive requirement occurs during SQL Server Reporting Services (SSRS) development wherein a client would request that a report parameter be configured to allow multiple values from a dataset that is populated by stored procedure, as illustrated in Figure 1.

Top 4 options for fixing Microsoft.AnalysisServices.AdomdClientUI.dll assembly error in SQL Server 2016

June 26, 2018 by Sifiso Ndlovu

I’ve always been in favor of an orthodox strategy when it comes to applying SQL Server updates which often goes like:

Instead of installing SQL Server Cumulative Updates, wait for release of service packs
When a service pack is released, install it in phases starting from the non-production environment (i.e. DEV, UAT) to eventually roll it out on production

Understanding benefits of Graph Databases over Relational Databases through Self-Joins in SQL Server

May 21, 2018 by Sifiso Ndlovu

Earlier this year, I published several articles on SQLShack with an aim of demonstrating tools available for visualising SQL Server 2017 graph databases. I was so caught up in the excitement of having SQL Server finally support graph databases that I forgot that some people still do not have a good grasp of how graph databases work let alone consider replacing their relational databases models in favour of graph. Although there are several ways that one can go about explaining the usefulness of graph databases over its relational counterpart, I have opted to focus on the benefits and strengths of graph databases by demonstrating the differences in which graph and relational databases deal with hierarchical datasets.

How to use Microsoft Flow to extract SharePoint Online list data

May 3, 2018 by Sifiso Ndlovu

Data extraction is a pivotal part of any business process particularly when it comes to running reports and facilitating business decision-making. In the article, How to configure OData SSIS Connection for SharePoint Online, I covered data extraction off a SharePoint Online list using SQL Server Integration Services (SSIS). You would have noticed in the aforementioned article that getting SSIS to successfully integrate with SharePoint Online lists can be a laborious exercise, especially if you haven’t installed the correct SharePoint SDK files. Thus, in business environments where business and Power Users have more control of data extraction processes, SSIS could get complicated for an ordinary business user to operate. Therefore, given the nature of our source data and the platform in which it resides, ETL architects and developers alike may need to find alternative ETL tools to SSIS. This brings me to Microsoft Flow which could be one possible alternative to using SSIS for data extraction. Microsoft Flow is part of Office 365 applications and just like SharePoint Online, is a cloud-based application that is freely available, easier to operate and effortlessly integrates with – amongst other applications – SharePoint Online. The aim of this article is to demonstrate the convenience of extracting data from one SharePoint Online list to another using Microsoft Flow.

How to configure OData SSIS Connection for SharePoint Online

April 13, 2018 by Sifiso Ndlovu

As data warehouse developers, we often have to extract data from a myriad of source systems. Thus, whilst some source systems readily integrate with our ETL tools there are instances whereby we need to install additional drivers and software addons in order to successfully connect and extract data from other source systems. Microsoft SharePoint Online is one such source system that I recently had to extract data from and its connectors are by default not part of the standard SQL Server Integration Services (SSIS) package template. As SSIS developers we often don’t have solid background on environments such as SharePoint, thus figuring out which version of SharePoint Software Development Kit to install in order to enable successful connection from SSIS can sometimes be a frustrating experience. In this article, I try to alleviate some of that frustration by sharing some of my recent experiences relating to getting data out of a SharePoint list using SSIS.

How to migrate SQL Server 2017 Master Data Services Models into another server

March 23, 2018 by Sifiso Ndlovu

Often as consultants, we don’t get to work onsite alongside our clients instead we are given copies of clients’ production environment and work on proposed solutions back at our offices. Once development has been completed, we then deploy and integrate our solution back to the client’s production environment. I’ve recently had to adopt a similar offsite development approach whilst working on a project that included development and configuration of master data services. In this article, I will demonstrate how a SQL Server 2017 Master Data Services (MDS) model can be exported from one environment (i.e. MDS Dev) and deployed into another environment (i.e. MDS Prod).

Replace bridge tables in a Data Warehouse with SQL Server 2017 graph database

March 8, 2018 by Sifiso Ndlovu

Just like in Santa’s Bag of Goodies, every release of SQL Server often has something for everyone – be it enhancements to DMVs for the DBAs, new functions for T-SQL developers or new SSIS control tasks for ETL developers. Likewise, the ability to effectively support many-to-many relationships type in SQL Graph has ensured that there is indeed something in it for the data warehouse developers in SQL Server 2017. In this article, we take you through the challenges of modelling many-to-many relationships in relational data warehouse environments and later demonstrate how data warehouse teams can take advantage of the many-to-many relationship feature in SQL Server 2017 Graph Database to effectively model and support their data warehouse solutions.

Impact of CLR Strict Security configuration setting in SQL Server 2017

February 13, 2018 by Sifiso Ndlovu

Every seasoned SQL Server developer will tell you that no matter how hard you try, there are just operations in SQL Server better implemented elsewhere than relying on native Transact-SQL language (T-SQL). Operations such as performing complex calculations, implementing regular expression checks and accessing external web service applications can easily lead to your SQL Server instance incurring significant performance overhead. Thankfully, through its common language runtime (CLR) feature, SQL Server provides developers with a platform to address some of the inconveniences of native T-SQL by supporting an import of assembly files produced from projects written in. Net programming languages (i.e. C#, VB.NET). I have personally found CLR to be very useful when it comes to splitting string characters into multiple delimited lines.

How to plot a SQL Server 2017 graph database using PowerBI

January 9, 2018 by Sifiso Ndlovu

In the article How to plot a SQL Server 2017 graph database using SQL Server R, I highlighted the lack of built-in graph data visualisation as one major limitation of the SQL Server 2017 graph database feature. In the same article, I went on to suggest making use of SQL Server R as one workaround that could be utilised in order to successfully plot and visualise diagrams out of SQL Server 2017 graph database objects. However, whilst 3^rd party graph database vendors such as Neo4j provide an interactive and hyperlinked graph diagrams that allows you to – amongst other things – easily drilldown and identify node-relationships as indicated in Figure 1, the graph plotted using SQL Server R is not very interactive in fact it is simply a static image file as shown in Figure 2.

How to plot a SQL Server 2017 graph database using SQL Server R

January 3, 2018 by Sifiso Ndlovu

A few years ago, one common business case I came across in my professional career that required modelling of data into a many-to-many entity relationship type was the representation of a consultants and their projects. Such a business case became a many-to-many entity relationship type because whilst each project can be undertaken by several consultants, consultants can in turn be involved in many different projects. When it came to storing such data in a relational database engine, it meant that we had to make use of bridging tables and also make use of several self-joins to successfully query the data.

How to enable and disable the Identity Cache in SQL Server 2017

December 20, 2017 by Sifiso Ndlovu

Every data warehouse developer is likely to appreciate the significance of having surrogate keys as part of derived fields in your facts and dimension tables. Surrogate keys make it easy to define constraints, create and maintain indexes, as well as define relationships between tables. This is where the Identity property in SQL Server becomes very useful because it allows us to automatically generate and increment our surrogate key values in data warehouse tables. Unfortunately, the generating and incrementing of surrogate keys in versions of SQL Server prior to SQL Server 2017 was at times challenging and inconsistent by causing huge gaps between identity values. In this article, we take a look at one improvement made in SQL Server 2017 to reduce the creation of gaps between identity values.

ETL optimization using SQL TRY Functions

October 24, 2017 by Sifiso Ndlovu

Introduction

An enterprise data warehouse ETL solution typically includes, amongst other steps, a data transformation step that converts source data from one data type into another. It is during this step that type conversion errors may occur and depending on the type of exception handling techniques implemented in the ETL solution (or lack thereof), frustration may occur for both ETL developers and DBAs when trying to identify and resolve type conversion errors. In this article we take a look at a trio of SQL TRY built-in functions that were introduced in SQL Server 2012, namely, TRY_PARSE, TRY_CAST, and TRY_CONVERT and how they could be utilized to reduce type conversion errors in ETL solutions and thereby saving developers needless troubleshooting exercise.

How to replace hardcoded lookups using SQL Server Master Data Services

September 14, 2017 by Sifiso Ndlovu

Introduction

A big part of the technical debt in my organization’s data warehouse (DW) and business intelligence (BI) environments relates to hardcoded lookup data. This is data required by the business to make sense of transactional data but was never planned for in the underlying source system and consequently get injected into DW and BI solutions. Inevitably, it is only a matter of time before DW and BI team lose track of the places wherein the hardcoded data reside thus making it difficult to maintain. Furthermore, due to lack of documentation or staff retention, anyone who subsequently takes over these DW/BI solutions can unknowingly create duplicate lookup data. In this article, I explain how we reduced such technical debt in my organization by moving most of the hardcoded lookups into SQL Server Master Data Services (MDS).

SQL replace: How to replace ASCII special characters in SQL Server

August 7, 2017 by Sifiso Ndlovu

One of the important steps in an ETL process involves the transformation of source data. This could involve looking up foreign keys, converting values from one data type into another, or simply conducting data clean-ups by removing trailing and leading spaces. One aspect of transforming source data that could get complicated relates to the removal of ASCII special characters such as new line characters and the horizontal tab. In this article, we take a look at some of the issues you are likely to encounter when cleaning up source data that contains ASCII special characters and we also look at the user-defined function that could be applied to successfully remove such characters.

Using SSIS ForEach Loop containers to process files in Date Order

August 3, 2017 by Sifiso Ndlovu

One positive thing to come out of my recent project that involved rewriting one of the Data Marts from our Data Warehouse environment was a confirmation of my suspicions with regards to the behavior of SQL Server Integration Services’ (SSIS) ForEach Loop Container. You see, I have long suspected that the ForEach File Enumerator type in SSIS’s ForEach Loop Container does not process time stamped text files in an order that could be deemed correct to the human eye. For instance, Figure 1 shows a list of text files containing data relating to Marital Statuses of FIFA 2016 Ballon D’Or nominees.

How to calculate work days and hours in SQL Server

June 26, 2017 by Sifiso Ndlovu

Like any other enterprise RDBMS system, SQL Server ships with several built-in functions that make developers’ T-SQL code clean, convenient and reusable. To demonstrate the efficiency of functions, say we needed to retrieve a server name for one of our SQL Server instances. Well, one of doing this would be to write a SELECT statement that would query the system view [sys].[servers] from the master database as shown in Script 1.