How to organize your business intelligence function (with examples)

In the age of data every corporation has in one form or another a business intelligence function. It is simply vital as having the traditional functions like accounting, finance, HR, sales etc. In essence "business intelligence" function handles the gathering, organization of data and delivery of necessary analytical content within your corporation. It is a central function that interacts with all other functions by feeding them the necessary analytical tools and resources. In more expanded view BI function also supports all of the enterprise systems with the help of IT department and outside contractors. Here are some examples of tasks performed by the BI function:

• create analytical dashboard on sales performance by account manager;

• create KPI dashboard about various corporate functions;

• support the enterprise accounting system;

• deliver data set to financial analyst on most recent paid invoices;

• and many many more...

As every other core corporate function the BI function is not directly revenue-generating but rather an important facilitator within the company. The value added is measured by the efficiency of data delivery and provided analytics. The efficiency is driven by the cost on one hand and then the utilization and impact of provided tools. Here are some examples:

• provides professional analytical tools to corporate functions, e.g. HR people don't have to prepare sophisticated dashboards, they can rely on the BI function;

• delivers fast and effective flow of data from systems to users, e.g. financial analysts don't have to waste time trying to compile list of all invoices, they can rely on the BI function;

• A lot more examples can be given. Here are 3 great articles on the value added of BI function within corporation:

  • https://www.techtarget.com/searchbusinessanalytics/tip/Top-benefits-of-business-intelligence-for-companies

  • https://www.domo.com/learn/article/how-business-intelligence-provides-value-to-companies

  • https://recoverydecisionscience.com/the-value-of-a-business-intelligence-team/

So lets take a look on how an BI should be organized based on our experience while working in this field.

General overview of BI structure

Lets start with the broad map of the BI function. In its most expanded version the BI function contains must cover the support of Enterprise systems, organization and delivery of data and creation of analytical toolboxes. Keeping this in mind here is how the structure would like:

Broad structure of BI function within Corporation

Everything starts with the Enterprise Systems Support team since 90% of the data used in the corporation will be coming from those core enterprise systems. Next we have a Data Engineering team that handles the second stage - grabbing and organizing the data. At the end of the data pipeline is the Analytics Center, or more traditionally known as the "narrow BI function" that has to turn that data into insights and analytical tools.

On top of those core teams, we must have a QA layer that is responsible for the quality of output, it is only logical as usually this is project-like work where the end deliverable must be checked. Also it is good idea to have a Project Management Layer that facilitates the different process and especially the relationship with end users (other corporate functions - Finance, HR, Sales, Procurement etc.).

This broad structure allows the cost-control, efficient delivery and scalability of the BI function. Although it is pretty wide and comprised of very different skill sets, it is generally recommended to have all of it under the same roof, not only because it is highly interconnected but also otherwise it is a huge risk for scalability. You can imagine data warehouse/lake team working on its own sets of project, that differ from the Analytics center..

In depth team analysis

After we got birds-eye view of the expanded BI function structure, now lets deep dive into each team's responsibilities, deliverables, risks and KPIs. Here we have flowchart of each team and the main roles that must be filled within that team:

Expanded view of BI function organization

So lets go one by one!

Enterprise Systems Support:

Mission: Support the existing enterprise systems like accounting, HR, Oracle/SAP modules etc. As well as required enhancements and further developments of these systems.

Roles: Most resource is spent by System Support Engineers that support the day-to-day functionality of those existing systems. However development resources must be kept in place for further enhancements and development.

Example of tasks: Troubleshooting billing systems; onboarding acquisitions into existing systems; development of journal upload portals

KPIs: Number of resolved tickets; Downtime of enterprise systems.

Data Engineering Team:

Mission: Collect and organize corporate data sets. As the Enterprise systems produce data, it is the job of DET to gather this data and organize it in Data warehouses, data lakes etc.

Roles: Needs to have data warehouse/data lake developers that are proficient into building resources and ingesting data. Also support members are needed to keep day-to-day operations. Data analysts prepare necessary data sets for the analysis by core BI teams and serve as connection.

Example of tasks: Ingest data set into data lake; resolve DQ issue in certain table; write SQL query requested by analytics center.

KPIs: DQ tickets; ingested data sets; time of delivery

Analytics Center:

Mission: Provide analytical tools for end users, work closely with stakeholders to obtain business objectives and use their mix of business and tech knowledge to deliver value.

Roles: Analytical centers are mainly comprised of BI analysts that must have good business knowledge as well technical skills (SQL, data analytics, BI platforms). Also for pure reporting it is possible to use less skilled reporting analysts. The BI platform support also resides within that team, because it needs to be in close contact to end users for organizing stuff like content, security models, access ect.

Example of tasks: Build analytical dashboards; deploy access security model; provide insights based on analysis

KPIs: End user feedback on provided analysis; BI platform tickets; number of successful deliveries.

QA team:

Mission: Standard QA work - provide analysis whether the finished deliverables (dashboards, data sets, analyses) meet the established standards and all requirements of the end users are fulfilled.

Roles: QA analysts that check each project.

Example of tasks: QA tests

KPIs: Number of tested projects

Project Management:

Mission: Facilitate the end-to-end execution of all projects. Help synchronize the work between different teams in the BI function. Be the bridge between the BI function and end users.

Roles: Project Manager

Example of tasks: Gather stakeholder requirements; Create and track schedule of project

KPIs: Number of successful projects deliveries; End user feedback on delivery

In some smaller and not fully developed teams some of those functions can become blurred. For example the QA work can be initially done by project managers if the capacity haven't gotten to a stage to require dedicated QA. Also some organization may for example move the support of the BI platform to the Enterprise Support Team.

Roles within each team

Next lets go even deeper and discuss the different roles that each of the teams must have. We are doing this discussion more from organizational point of view with emphasize on what will be their responsibility, what skills to look after, where the highest cost must be allocated and what is proper size of the teams.

Obviously there is a ton of difference between corporations, and the definition of roles, teams etc. can vary wildly. Nonetheless lets review the backbone case here:

Role details within BI function

Lets start with the two support layers - QA and Project Management, because as stated before it is quite possible these two can be mixed into a single team in order to cut costs.

Usually the QA analyst is relatively expensive role as it requires technical skills. In the context of the BI function within corporate (rather than client facing) environment the QA is quite often mixed responsibility of different parties involved, but generally it is a good idea to invest in a QA-dedicated unit of 2-3 people, especially when the BI function begins to scale up.

Key supplementary unit is the Project Management (PM) function. In not so developed BI functions the PM is often the responsibilities of the core teams. Basically BI analysts handle themselves the gathering of requirements and general communication with stakeholder. As the BI function matures it is usually normal the PM role to emerge. It doesn't have to be big investment, junior and mid-level people will suffice as PMs as it doesn't really require tech skills.

Going into the core teams we start the Enterprise System Support team. It is closest structure to true IT development team. As usual the supporting role (IT Support Engineer) should be the least expensive in relation to the core System/Development Engineer. The latter has the responsibility not only to support and troubleshoot existing systems but to develop enhancements and potentially all new systems, this is a role demanding software development skills. Also as the BI function scales up and matures it is also good idea to split the DevOps function into a separate role, which also will be in the expensive category, because it is better to keep it as small team highly professional rather than balloon the headcount.

The most expensive role within the Data Engineering Team should be the DWH/DL (data warehouse/data lake) engineer that is responsible to develop the chosen DWH/DL, ingest data and generally keep the house clean. You do not want to save money exactly from that central role, because this is the bridge between software (enterprise systems)-oriented piece and the data infrastructure and analysis. Usually those should be very few engineers with exceptional tech knowledge. Around them should be the DWH/DL Support Engineer that helps with maintaining the existing data assets. It is less expensive and little bigger team so that it can cover different data warehouses and lakes throughout the organization. As the function matures also there should be Data Analyst position that uses the existing data assets and provides curation to the BI analysts downstream.

Finally the biggest team should be the Analytics center, or traditionally called BI team. It is client-facing team so it should be able to serve multiple purposes. By far the most important role is the BI Analyst, that uses the existing data assets (lakes, warehouses) to provide the analytical tools to end users. It is a blend role - tech savvy to work with data assets, very proficient with using the chosen BI platform (Tableau, Power BI, Looker etc.) and at the same time has deep business knowledge to provide the analysis to end users. It is the bridge between the entire BI function and end users. Also I would advice to separate the BI Platform Admin in a separate role, to help them focus on maintaining the servers, security models, track usage etc. As the BI function matures also Data Scientists are added, those have the deep statistical and math knowledge and help the BI analysts in deploying more complex models as machine learning, AI etc.

The BI Funnel

Overall the size of the the BI function can be described as a funnel facing the end users:

BI function funnel

It is following the flow of data from the core enterprise systems , through the data warehouse/lake and all the way to the dashboards and analytics that the end users use. At the bottom of the funnel is the mostly IT work, and as we go up the required tech skills turn more and more to business knowledge and analytics. In general we begin with small team as the data is just data in some systems, as it begins to take shape of analysis we expand the teams to cover different data domains (sales, finance, HR etc.). This also allows scalability as the core of the funnels stays small even though the data grows exponentially, and we just have to scale up the Analytics Center to encompass the new domains. At the same if properly centralized we maintain control over the entire "data process".

Conclusion

Every corporation has BI function even the start ups, the flow of data from systems to managers and decision makers is vital to company's success. As the corporation starts to manage and organize the BI function it will reap the benefits. Properly organized function can add enormous value, while a poorly managed will drain resources for marginal gains.