MongoDB Connector for Splunk for Optimized Data Processing at A1 Bulgaria

Decentralized data storage and no real-time visibility of crucial systems and events; Time-consuming incident management and troubleshooting, taking a lot of unwarranted effort.

Telecoms have immense infrastructures, various, complex systems, respectively large amounts of raw data distributed across the enterprise. Analyzing the performance of those systems can be challenging and there is a need for a flexible, real-time monitoring solution that aggregates and filters out data, ensuring enterprise-wide data delivery.

A1 Bulgaria has adopted the Splunk platform to provide visibility into key business areas, including IT Ops, Security Analytics, Compliance, Finance, Fraud Management. With the implementation of the growing number of uses cases on the platform, one of the biggest pains was the lack of end-to-end process visibility and a significant amount of time and effort lost in processing data and troubleshooting.

A1 Bulgaria uses an enterprise ESB platform, which centralizes the data flow and provisions the communications between all the systems in the organization. The monitoring of communication between the individual systems is extremely hard due to the different programming languages and technologies they are built on. Each system logs differ by its format, that is why troubleshooting could be challenging and time-consuming, especially if it is the duty of one person.

The enterprise ESB platform combines over 400 applications that record logs in one joint database. This database has limited logging, and the data it aggregates is not enough to carry out end-to-end process monitoring. In addition, if an error occurs, it is hard or almost impossible to identify where the problem is and which system is causing it.

And here comes Splunk – combining all data into one single format and providing a complete view of all organization’s systems, devices, and interactions. But how we transfer all the data we need from the organization’s infrastructure into Splunk?

Using MongoDB to aggregate and store data; Transition and visualize this data into Splunk through modular inputs, ensuring performance, flexibility, speed, and stability; Utilize the maximum potential of the Splunk instance while keeping an eye on the data that matters most.

To enable business managers and executives to get the full picture of the system’s state and health, the current database could not be used. BRIGHT’s consulting team came up with a completely new solution.

BRIGHT helped A1 rearrange the infrastructure, integrating a MongoDB database to aggregate all the data coming from the adopted systems – without missing a log or thinking about data modelling. MongoDB is fully optimized to support large amounts of structured and unstructured data, providing flexible and cost-effective data storage.

Once all the data needed was stored in MongoDB, the team had to ingest it into Splunk. The Splunk MongoDB Connector that was used solved the data integration challenges with the following key capabilities:

• Modular inputs now run on a specified interval and due to adopting a multi-processing approach, the app can quickly ingest large amounts of data;
• More than 80% performance increase by multithreading, executing queries almost eight times faster;
• Integrated flexible payload filtering which eliminates vast amounts of unnecessary data. In the world of ‘Data to Everything’, our approach allowed A1 Bulgaria to get only the data they actually need, critical for their operations;
• The check-pointing mechanism can ingest data, which could have been missed due to various reasons, so no time is lost in unnecessary restarts.

BRIGHT’s role as an A1 Bulgaria Splunk consulting and implementation partner helped the company find an elegant way to take the stored raw data and transfer it to people, empowering the organization with an end-to-end monitoring and continuous improvement of delivery.

The most significant value for A1 is the ability to perform end-to-end monitoring and optimize the production system’s performance. Now that the whole infrastructure is connected to Splunk, the IT manager responsible for the production environment can track the communication between systems in real-time. For example, he could see all requests from web service to provisioning service for the last 60 minutes and identify how long their processing was. He is able to identify any delays or errors and know where to optimize. It is now clear how each system works and its health can be easily monitored. This can eliminate the backlog and mitigate the noise in finding exactly where the issue is.

Overall, BRIGHT MongoDB Connector for Splunk app provided a seamless, flexible, and quick connection between MongoDB and Splunk.