
Apache Spark Application and Vector Database Build a Product Recommendation System (Recommendation Engine) Increase Cross-Sales Revenue by 45%
Project context and Bottleneck in cross-selling strategy: The difference between a survival sales website and an e-commerce empire (like Amazon or Shopee) does not lie in the number of products they sell, but in the ability to understand what customers want to buy before they themselves realize it. This Case Study project delves into the technical restructuring of a digital electronics and home appliance retail system with more than 200,000 product codes (SKUs) and about 3 million visits per day.
Before the project began, the business' cross-selling and up-selling strategy depended entirely on a tedious static "Related Products" system that was manually configured public in the source code. If a customer looks at an iPhone, the system will display naive suggestions for other iPhone models, instead of displaying cases, fast chargers or wireless headphones. The old relational database (SQL) system could not handle queries analyzing the purchase history behavior of millions of accounts. As a result, the add-to-cart rate from the "Suggested Products" block (Click-Through Rate - CTR) bottomed out below 1.2%, and the Average Value per Order (AOV - Average Order Value) could not grow even though the advertising budget poured into the homepage was very large. An urgent requirement is posed: Must build a 100% automated Recommendation Engine, capable of Real-time Hyper-personalization prediction.
Technical Architecture 1: Batch Big Data Processing with Apache Spark: In order for the Machine Learning machine to reason accurately, it needs to be "eaten" High quality data. The Data Engineers team built a Data Lake on the Object Storage platform (similar to Amazon S3) to collect all raw data: From clickstream history, Dwell time, search history, to transaction history and star reviews of 5 million customers over the past 3 years.
Then, the distributed computing engine Apache Spark was put into operation. With the ability to split parallel processing tasks across dozens of cluster servers right on RAM, Spark processes Terabytes of log data every night in less than 30 minutes. The Collaborative Filtering algorithm based on Matrix Factorization (ALS) is applied here. Spark will analyze and find hidden similarities between customers: "Users A and B have never known each other, but bought Samsung phones and LG TVs together. Today user A just bought a Soundbar, the system will predict an extremely high probability that user B will also need that speaker system, even though they have never searched."
Technical Architecture 2: Real-time semantic search using Vector Database: However, The Collaborative Filtering algorithm is only effective on customers with a long history. For guest users accessing for the first time (Cold Start Problem), the system needs a different mechanism. That's where Vector Database like Milvus or Elasticsearch (with Dense Vector field) comes into play.
The Artificial Intelligence (AI) system will read the entire text description, specifications and images of 200,000 products, then "Embedding" them into arithmetic strings of multidimensional spatial coordinates (Vectors). Each click action of a guest user (Guest) on the website will immediately be updated into a Behavior Vector (User Profile Vector) on Redis high-speed cache. When the customer surfs to the bottom of the website, the Frontend will fire the API back to the server. The Vector database will perform the Cosine Similarity or K-Nearest Neighbors (KNN) calculation to find the 10 products whose direction vector is closest to that user's behavior vector in the space of millions of data points. The entire process of scanning and responding to this hyper-personalized suggestion list takes place in a mere 20 milliseconds (0.02 seconds) – a speed unimaginable compared to traditional database LIKE '%keyword%' queries.
Value Contribution and Shocking Business Results: After 3 months of running A/B Testing – in which customer group A displays suggestions old ideas, group B displays suggestions using the new Recommendation Engine - the results returned have completely changed the business landscape of the enterprise. The new system not only recommends the correct compatible accessories (For example, suggests the correct USB-C charging cable code for phones that support Type-C port, but does not suggest Lightning cables), but also knows how to adjust the suggested price range based on the "spending" level of each customer file.
The click-through rate (CTR) in the "You might like" area skyrocketed to 18%. Most importantly, revenue from cross-selling items - which are accessory products that bring the highest profit margin for the company - has grew 45%. Average cart value (AOV) increased by 30% as customers continuously picked up small items that the system called for so well. This Case Study is the most eloquent proof of the fact that: In modern e-commerce, behavioral data is invaluable oil, and the combination of Big Data architecture (Apache Spark) and AI (Vector Search) is the oil filtering machine that brings huge revenue streams without spending a single penny on advertising costs.