Revolutionizing Product Information Data Management with AWS

BSH is one of the world’s leading home appliance manufacturing companies, with 40+ production sites in Europe, the USA, Latin America, and Asia along with a strong network of 80 sales, production, and service companies in 50+ countries. With outstanding home appliances and useful digital services, BSH meets the needs of consumers worldwide.

The goal was to develop BSH’s PIM (Product Information Management) processes so that BSH’s end users could easily access up-to-date product information from different points and on different applications. Within the scope of the PIM application programming interface (API) system, a wide range of information, including the features and documentation of BSH products, can now be queried in several languages and queried different kind of data type from different endpoints. The application required the establishment of a high-performance and scalable architecture that could handle simultaneous systems queries and results in a short timeframe in a secure way. API consumers can be too various from different countries. API must be publicly accessible, and we need to manager permission who can access this API.

The Commencis Cloud Team developed end-to-end applications, conducted performance and load tests, and utilized multiple AWS services in the solution. The solution consists of two main parts: Production Information Data Get Services & Product Information Data Collection Services.

Product Information Data Get Services are set of services running as Lambda functions behind API Gateway to return up-to-date product information data that is stored Aurora PostgreSQL. To be able to meet high-performance requirements, the product data is read from the read replica instance of Aurora PostgreSQL. These Lambda functions are called from various intranet and internet applications of BSH through the API Gateway which is secured by API keys.

Endpoints:

• Multiple products – /products
• Single products – /product/{id}
• Single Product Variant – /product-variants/{id}
• Compare Products – /compare
• Product Families – /product-families
• Sales Programs – /sales-programs
• Swagger Doc – /docs

Product Information Data Collection Services are set of services running as Lambda functions to collect BSH’s consumer product information in XML format and store the data in database. On-premises applications that are the master of such product information data send messages about changes (new product, update product, delete product) in consumer products and put the associated hierarchical product data in XML formatted files into S3 bucket. A Lambda function listening the SQS, is triggered automatically when a new message is sent, reads the files in XML format stored in S3. After parsing and converting the files, the content is stored or deleted in the Aurora PostgreSQL database’s production information tables. The database operation triggers another Lambda function which pushes notifications about these database operations to SNS. Application that are subscribed to SNS topics are being notified about product information data change, which will allow them to call Production Information Data Get Services to get up-to-date product information data and to update the product information data in their systems.

Bitbucket Cloud: Bitbucket Cloud product is used to store source codes of the Lambda functions, release management and deployment of the new versions and fixes.
JMeter: JMeter is used to test and report the performance of the Product Information Data Get Services running as Lambda functions.
Terraform: Terraform scripts are developed to define and maintain the AWS infrastructure. Scripts are stored in Bitbucket Cloud like Lambda functions and Api gateway source codes.

API Gateway: Lambda functions for Product Information Data Get Services, are running behind API Gateway and secured with API key.
Lambda: Lambda functions are developed in GoLang to meet performance requirements for both Product Information Data Collection Services and Product Information Data Get Services.
Aurora PostgreSQL: The product information data is stored in Aurora tables. Aurora is deployed as previsioned, still Serverless v2 deployment is on the roadmap to meet both high performance requirements and unpredicted traffic pattern. Aurora read replica serves for Product Information Data Get Services.
SQS: The service is selected for decoupling master product datastore in on-prem and its replica product datastore in AWS. When the PIM is offline due to maintenance, version update, etc. SQS is online and continues to collect product information change messages.
SNS: The service is used to publish notifications when the product data store gets updates.
S3: XML files having product information are stored in S3 bucket.

BSH now has a product information store in AWS which is running in parallel to the master product information database and being updated asynchronously. The product information data store can be queried concurrently by many external systems, deployed in high-performing and highly scalable architecture at AWS.

The solution is planned to be evolved to store other BSH data on the same highly scalable architecture. AWS CloudFront is planned to be installed through the AWS API Gateway, as the PIM API system needs to be queried from different countries in the later phases of the project. This will ensure that delays caused by the data connection at remote points are avoided.

Product information (new, update, delete) data is created in on-premises applications, stored in S3 bucket as XML formatted files, event message is put in SQS FIFO in parallel. Event message put in SQS triggers Lambda function to convert and store the product information data in Aurora database. Database operations event message is put into SNS with another Lambda function. External applications that are subscribed to SNS topic are being notified, which triggers external application to query up to data product information data either individually or as bulk with the API exposed via API Gateway in front of Lambda functions.

Expose APIs for internal and external applications in a secure way, which will return up to date BSH multilingual product information data with product document files, either individually or as bulk (up to 20 products in each API call) in JSON format. The APIs should meet the following performance requirements:
1000 concurrent call per second (in average) of API which will return data
P50 < 200ms
P90 < 400ms
P95 < 500ms
Number of concurrent API calls can reach up to 5000 requests per sec with response times increasing linearly.