Dhananjay Buva

Topic: Case study for holistic approach to IoT testing

Abstract: Use Case:
One of our clients is esteemed provider of a big cloud platform as a service. The client needs to fetch lots of metrics for variety of business needs. This data is then used by the client itself and customers of clients as well to gain lots of insights.

The cloud implementations of the client are of tremendous size and span across different business models, domains and use cases. As a result, on daily basis, millions of records in millions of tables get processed and many users use the platform extensively. And the metrics that are captured on top of this have a wide variety, velocity and volume. Building and testing this entire capability was big project.

Few of the challenges faced during this project include:
1. Early detection and correction of errors/discrepancies in data.
2. The variety, volume and velocity coupled necessitated need of almost 100% very fast paced automated testing. The frequencies at which metric get captured are different, that makes this even more challenging.
3. Building trust with users of this data who have wide variety of profiles including data scientists, finance, corporates, sales and marketing etc.
4. The familiarity with client’s platform was limited considering its size and variety.
5. The mainstream business usage needs to remain unaffected while all these metrics get captured or data for these metrics gets generated.
6. Many new metrics need to be captured on recurring basis based on insights generated from earlier data.
7. In many cases, it was necessary to make these metric backward/forward compatible.

Solution:
Holistic approach was followed to develop and test this capability since this was not fitting traditional methods of software development, testing and processes. Both functional and non-functional aspects of the testing were covered.

On testing front, initially semi-automated / automated testing frameworks were developed that consisted of excel, database queries and methods to generate variety of test data. This was mainly used to validate accuracy of the data and functional aspects of the required capability.

Another python based big data framework was developed which used various techniques to read the necessary data volumes, categorize the metrics, perform statistical analysis on them and report the discrepancies/errors in data at very fast pace. Visualizations were built on top of them to instil trust in the results.
The results of this helped develop and test further enhancements to get correct data in case of discrepancies.
This was coupled with change in processes to deliver new metrics daily. Normally the enterprise applications have release recycles in the range of months while this capability necessitated almost daily releases as needed. The data pipelines and CI-CD pipelines were developed to support this, and they are tested using automation utilities and canary testing mechanisms.
Performance of the capability was extensively validated using heavy volumes of data that guaranteed smoother functioning of mainstream platform functionality during data generation/collection phase.

Required penetration tests were performed on data fetch mechanisms.
All this part so far was mainly related with data generation, cleansing and integrity validation phases. Equally important areas are the data ingestion and storage. The entire metrics reside in a huge warehouse and data warehouse testing was also big part of the project. This involved verifying Datawarehouse protection, alerts, ETL capabilities, upgrades, archival and disaster recovery mechanisms.

The holistic approach helped us detect many issues prior to the production rollout. The project is operational in production since almost 15 months and has a defect leakage ratio of < 1%. Thanks to the entire development and testing teams involved in this project for tremendous effort.