Forge Case Study
Revolutionizing Recommendation Engines for Wallets
Project Overview
Our latest initiative at Tempered Ai has culminated in a truly transformative accomplishment: the successful development and deployment of a real-time recommendation engine tailored to the intricacies of token and NFT wallets. With a powerful suite of Google Cloud technologies — BigQuery, Dataproc, Cloud Functions, and Vertex AI Matching Engine — we’ve engineered an advanced system poised to manage the complexities of approximately 65 million wallets, each enriched with over 20,000 unique features.
Illustrating Precision with K-Nearest Neighbors (KNN)
This project isn’t just about managing large datasets; it’s about intelligently connecting each wallet to its most relevant neighbors in the vast universe of tokens and NFTs. The images below showcase the K-Nearest Neighbors algorithm in action:
Here, we highlight the engine's capability to discern and group wallets based on token similarities, fostering precise, targeted recommendations.
As our system zeroes in on a singular wallet, it dynamically identifies and maps the nearest neighbors, illustrating the power of real-time analytics.
The third image encapsulates the overarching reach of our recommendation engine, displaying the extensive network and the pinpoint accuracy of our KNN approach.
The Challenge
The digital wallet space is rapidly evolving, with users increasingly seeking personalized recommendations that align with their interests and holdings. Addressing this need required an innovative approach to process and analyze vast amounts of data efficiently, ensuring recommendations are both relevant and timely.
The Solution
Our solution involved the creation of a ScaNN index, meticulously designed to process data from wallets, interests, tokens, and NFT holdings. This index, once deployed as a model, enables us to query wallet IDs, returning the nearest neighbors for each – a process that allows for the dynamic generation of wider, yet highly targeted, audiences based on predefined criteria.
Technical Execution
- Architecture and Data Processing: We initiated the project using Dataproc for handling large datasets and Vertex AI for index creation and updates. Key steps included setting up a VPC network, storage buckets, and enabling necessary APIs.
- PCA Models: Our approach to data normalization and PCA models involved both standard and sparse matrices, ensuring optimal representation for tokens and NFTs.
- Deployment and Accessibility: The ScaNN index was deployed seamlessly, with updates facilitated through various interfaces including UI, CLI, and workbench. Access to the model was achieved through a cloud function, integrating directly with BQ queries for real-time execution.
Results and Impact
The deployment of this recommendation engine marks a significant milestone in personalized marketing for wallet users. Our system adeptly utilizes a pipeline that commences with BigQuery, harnessing its vast data processing capability to feed into PCA models. These models are intricately refined in our PCA Notebook to distill the essence into Reduced Features, ensuring that each wallet’s unique data footprint is captured with precision.
Subsequently, the Vector Index translates these features into a deployable framework, leading to the deployed model that sits at the core of our recommendation engine. This robust system meticulously identifies All Nearest Neighbours, pinpointing precise wallet cohorts and culminating in the Target Group, which forms the basis for bespoke marketing strategies.
Technical Innovations and Learnings
By integrating complex data processing with smart modeling, we have crafted a tool that not only predicts but also informs targeted engagements at scale. This is more than an engineering feat—it's a leap towards a future where marketing is driven by deep data insight.
- Sparse Vectors Challenge: Addressing the issue of skewed recommendations towards high-value portfolios involved deep dives into PCA feature distributions and adjusting distance calculations.
- Scaling with Spark: Transforming proof of concept code to Spark enabled handling large datasets effectively, showcasing the power of collaboration and technical flexibility.
- End-to-End Verification: A critical phase involves processing a sample dataset to ensure accuracy before scaling up, highlighting the importance of thorough testing and validation.
Future Directions
Looking ahead, we aim to expand our wallet database, refine our recommendation algorithms, and explore automated updates to further enhance our engine’s performance. Additionally, considerations are being made to integrate more nuanced filter options and leverage real-time data for even more personalized recommendations.
Collaborative Success
This project exemplifies the synergy between our team and external partners. The collective effort and shared expertise have not only led to the successful realization of this project but have also set the stage for future innovations in the realm of digital wallet services.
This case study underscores our commitment to technological excellence and our ongoing journey to forge new paths in the digital landscape.