What is Microservice Reporting?

Microservice reporting is when reports are created from a microservices architecture. While the microservices architecture offers many benefits to organization, it creates challenges for reporting due to its fragmented nature.

What is a Microservice Model?

To understand microservice reporting, there must be a good understanding of the architecture behind it. A microservice architecture is a cloud-based system where an organization’s software system is made up of a number of smaller components. These services or components are independently deployable, but are loosely linked together, generally using application programming interfaces (APIs). Similar to how a brain operates, the services are like brain cells, linked together by synapses (APIs) which allow the communication of information from one cell to another. Each of these microservices are responsible for a specific action or requirement, and they automatically deploy when needed.

These services all have their own databases, and do not share information from a central database the way other systems do. This is referred to as being ‘bounded,’ with everything needed for this service to operate bound up in the app.

This bounded nature is important to ensure there is no unintentional coupling of services. If data schemas are shared across the services, a change in one must be replicated across all the underlying data schemas. By keeping everything separate, this means there are limited changes, full autonomy of the services, and that the architecture retains its full agility.

Organizations choose to use microservices for a variety of reasons. The independent apps can easily be updated and changed, and new features can be added quickly. Different teams can operate different stacks of apps or programs that operate under the one umbrella, adding enormous agility to the business model.

It can also be a very cost-effective architecture. APIs are free or low cost; individual programs can be software-as-a-service (SaaS) and user-pays. It is entirely scalable, and there is no huge infrastructure requirement.

Businesses that use microservices architecture include Amazon, Netflix, Uber, and SoundCloud. It is truly a software structure that allows enormous flexibility and scalability in a business model.

However, the disadvantages of microservices are that the APIs create more risk for cyberattacks and the diversity of programming language and communication between the apps. But the biggest issue relates to reporting. Each app or program has its own database. So, how does a user pull all the data from the databases to run reports? This concept of pulling data from each application does not preserve that core value of bounded context.

Microservice Reporting

If an organization is running a microservice architecture, a simple model could assume that it has different programs for:

• Finances and billing
• Workflow tool
• Sales program
• Payroll system

Each of these microservices has its own database. APIs work to link these programs together so they can operate under one interface. However, the problems start when reports are required that pull data from multiple databases.

An example is when wages are based on completed and billed jobs. The finance tool needs access to the workflow tool, and only then does the payroll tool have accurate information in order to pay employees. If a business wants to run a report showing work in progress versus completed sales for the month, it needs to access three different programs.

It becomes even more complex when there are many, many microservices. Netflix, for example, has more than two billion API edge requests per day, spread over their 700-plus microservices. Uber uses over 1300 microservices. With a vast number of services, reporting becomes more complex.

Challenges with Microservice Reporting

There are three main challenges with microservice reporting.

Performance Issues

If a company has two billion API requests a day for business functions, imagine how much reporting will add to the load. If a report tool is pulling data from multiple sources, often several hundred times a day, how is this going to affect the system? It’s going to slow processes down considerably, or possibly end up timing processes out.

Anti-Pattern

The very idea of microservice architecture is that every service is bounded. That is, each application is a mini-program in itself. It contains everything it needs and operates within its own boundary. It doesn’t need access to anything else in order to complete its functions.

When reporting requires that services share their databases, it becomes anti-pattern.

Different Data Models and Types

If each service is written in a different programming language, there needs to be an extra layer of communication between the services to allow comprehension. And then, if the way the data is presented isn’t in the same form as another, it needs to be automatically manipulated so that it is able to be digested and used by the reporting tool.

Conflicting Data

Because each service has its own database, there may be data redundancy, duplication, or conflicts. For example, if a vendor is also a customer, and they have the same physical address but one is incorrect in the system, this could cause problems in data compiling.

What Reporting for Microservices Requires

Good reporting is essential to the health and growth of an organization. A good reporting system needs to have the following elements:

Easy to Use

Reports must be simple to run. Using a central dashboard, reports should be easy to select parameters and apply the relevant data. Because microservices all fall under the umbrella of a dashboard, this is achieved easily.

Single Source of Truth

For reports to be accurate, the data needs to be correct. There should be a single source of truth for data, with no double-ups, old, or incorrect information, and everything that the report needs should be there. This is the biggest challenge for microservices since the applications are separate and loosely coupled.

Timely

In a world of instant everything, reporting needs to be up-to-the-minute. When there are holiday sales, a change in working environment, or a different customer behavior, these changes need to be noted and addressed as soon as possible. This also presents challenges to the microservices model, depending on how data is accessed.

Fast

If a report takes hours to run and places strain on resources, it’s a significant barrier to successful reporting systems. Running thousands of APIs to access data simultaneously creates a considerable barrier to good reporting in microservices architecture.

Automation

The system of obtaining the data, extracting it, processing it, and running the reports should be highly automated. Users should not have to access the back-end, run manual operations, or perform manual manipulations to data.

Easy to Understand

Presenting reports in rows of granular data is not always what is needed, nor is it the best way for information to be understood. Instead, in order for reporting to be easily digested by a range of users, it should be presented in graphs, tables, and graphics that quickly show trends, patterns, and figures.

The Ability to Drill-Down

If a user or presenter has a simple graph but needs the ability to understand how and why something happened, there needs to be functionality to allow users to see granular data. Organizations need to be able to easily drill down into information to identify problems, strengths, and anomalies.

Artificial Intelligence (AI) Functionality

There is no point in having data if an organization can’t create actionable improvements based on it. To enable this function, there needs to be AI functionality so that data becomes insights and predictions. Once data is collected, adding AI functionality is easy given the segmented nature of microservices.

Potential Solutions for Microservice Reporting

There are a number of options available to allow reporting in a microservices architecture. The correct solution for an organization will depend on many different factors, and the answer is not one-size-fits-all.

The first step is finding the correct services. The model of microservices means that each task requires a different app or program. These microservice will sit alongside all the other apps and programs. It is likely that a business will need a service that pulls or receives data, another one with reporting functions, and then another with AI capabilities.

Once these services are installed, it’s a matter of pulling the data from the other microservices and creating the required reports.

Direct Database Connections

The reporting service directly connects with the other microservices, retrieves the relevant information from their databases, and funnels it to a data warehouse. This is simple but means any change to a database schema will require an additional change in the reporting service, too. It also breaks the bounded context that is so important in the microservices architecture.

Aggregation Service HTTP Pull Model

This service uses APIs to connect with the relevant microservices. It calls up the information required and funnels it through the data warehouse and reporting tool. However, this method is prone to slow performance and 504 timeout errors. Each additional service it needs to talk to adds more load to the system, aggregating multiple data sources that create massive performance issues.

Once again, because this is a pull model, it violates the bounded context.

Batch Pull and Dedicated Database

In this strategy, a command is executed on a schedule and information is pulled in batches from all the relevant microservice databases. This data is then bulk updated into the reporting database. However, as with the direct database connection, any change in schema could break the entire batch job. Every time the schema changes, the batch functions need to be updated too.

The other problem is that as a pull model, it is contrary to the bounded context, and it may not result in timely reporting. If the pull request is tied for when the load is lightest, at 2:00 AM, that’s only one update per day.

Dedicated Database with Asynchronous Event Pushing

This is more complicated, but it yields better results and does not violate the bounded model. An event processor microservice receives updates when anything in another service updates or changes. The app, rather than having information pulled from it, pushes event notifications out. That means a data capture service is notified of any relevant updates, and the database is updated. From there, a reporting service can run reports as needed.

The benefits of this method of reporting include:

• Preserving the principles of microservices bounded architecture. The reporting service is not listening to the apps or pulling information.
• Ensuring the resulting reports are up to date because the reporting service is notified of events or information updates.
• Filtering out information that is not required throughout the data capture service.
• Aggregating data into a form that the reporting service can utilize through the data capture service.
• Providing asynchronous reporting to avoid heavy processing loads, timing out, or temporal coupling.

How to Resolve Conflicting Data

Because there could literally be thousands of databases, it’s almost certain that there will be double-ups of data, or incorrect or flawed data. Resolving this is important to creating accurate reports. There are a couple of different solutions to this challenge.

A Service to Update Databases

If there is an intermediary service that recognizes when a piece of information is updated, it can push that update through to the other services. For instance, if a physical address of a customer was updated, that information is pushed through to all instances of that customer’s information. This can be incredibly complex and require more bandwidth in the system, and it is anti-pattern.

Choose a Single Source of Truth for Each Data Type

Because each service in this architecture performs a single task, the database for it does not need to be comprehensive. The billing department needs to know the postal address, while the sales department needs the physical address to ship the product. Nominate a service to be the single source of truth for that piece of data.

Microservice Reporting Requires Planning

There are a number of challenges for microservice reporting, and it is not the simplest architecture in terms of reporting. Even maintaining consistent data requires more planning and thought than other traditional architectures. However, it’s possible to run reports while still upholding the principles of bounded services.

If an organization wants to adopt a microservice framework, the IT team needs to put in considerable planning and groundwork to ensure that data consistency and accuracy come first, followed by a focus on the route of data pushed from source services to other services that create data lakes as a source of data, then processing that data to become the required reports.

Considering the huge benefits that the microservice model can offer an organization, it is worth the extra complexity in reporting. All the challenges can be overcome with thought, and planning.

Related Resources