Eva Giannatou, Or Levkovich, Martin Tegner, Sakis Athanasiadis, Vanessa Ho, Isabella Amorim, Patrick Miller, Nirmal Kumar Rajendran

Session-based personalized campaign recommendation

Personalization of marketing communications is a challenging task. At IKEA, customers have diverse preferences and needs as they are in different phases of their purchase journeys and their interests are spread across various product areas. Therefore, there is a high value-potential if we can anticipate the next-best marketing campaign and deliver personalized recommendations at scale.

We present an approach for recommending digital marketing campaigns. We use a pre-trained LLM for encoding multilingual email-text as contextual input and a transformer for modelling sequential customer events. We model customer interactions with email programs and web actions as multiple learning objectives, using customer session data and email features as inputs. The goal of this setup is to personalize campaign recommendations, create targeted audiences, and estimate the performance of new campaigns.

Previous work

Effective personalization in recommender systems relies on capturing user intent, preferences, and context. A common approach to model these aspects involves techniques from natural language processing, such as Word2Vec (Mikolov et al. 2013; Hidasi et al., 2016), to represent customer interactions in a continuous vector space. This is based on the rationale that “meaning“ — in the form of semantic relationships — will be encoded within the vector space during learning.

Word2Vec typically uses a fixed context window, which may overlook dynamic patterns such as the recency of user actions. Sequence-based recommenders address this limitation by modeling both long-term and short-term dependencies using techniques like LSTMs and attention mechanisms (Liu et al., 2018). The transformer architecture (Vaswani et al. 2017) further advances the field due to its ability to handle long-range dependencies, making it suitable for recommendation systems (Fang, 2021; Kang and McAuley, 2018; Sun et al., 2019a; Sun et al, 2019b). Transformers4Rec (de Souza Pereira Moreira et al. 2021), powered with HuggingFace, exemplifies this application for session-based recommendations.

Large Language Models (LLMs) are now common in recommender systems, serving as both direct recommenders and feature extractors for context-aware embeddings (Wu et al. 2021; Wu et al. 2023; Xiao et al. 2022; Zhao et al. 2024). In this work, we employ BLOOM (Le Scao et al. 2023), a large transformer model trained on a diverse corpus of 46 languages and 13 programming languages, to derive multilingual email text embeddings from email subjects, thereby increasing the context sensitivity of our model.

Multi-task learning is commonly used to optimize various customer behaviours simultaneously and to improve model generalization (Gao et al. 2019; Hadash, Sar Shalom and Osadchy, 2018; Meng, Yang and Xiao, 2020; Ruder, 2017; Yang et al. 2021). Employing multiple prediction tasks and sharing parameters between them further acts as a regularisation factor, and lead to enhanced generalization across different tasks.

To build a personalized email campaign recommender we used an instance of Transformers4Rec with XLNet (Yang et al. 2020) where we adapted the model architecture, inputs, and prediction tasks to cater to email campaign prediction. The model is modified to accommodate multimodal inputs, where sequential session data (categorical, item features) and customer context (categorical and continuous user features) are encoded with XLNet embedding layer to learn the contextual representation of session actions. Additionally, we integrate extra input features comprising embeddings of email subjects’ text, generated through a pre-trained Large Language Model. The model’s output is also modified to accommodate for multi-task predictions, including both a next-item prediction head and a regression head. By adapting Transformer-based models with multi-task capabilities and enhanced input features, we achieve a robust and contextually aware recommendation framework.

Data

Customer preferences are modeled using temporal and sequential web events data, collected with user consent. This dataset includes interactions with products and online services, such as user searches, items viewed, cart activities, inspirational content, home-furnishing advice, and interactive studio-planners, such as for kitchens or cabinets. Additionally, we also include other details about customer interactions, including the type of page viewed, product category, and the time elapsed since the previous web interaction.

Customer preferences for email campaign programs are modeled using non-temporal tabular data on interactions with previously sent email campaigns. We consider five types of potential interactions [CLICK, OPEN, COMPLAINT, OPT-OUT, NO-ACTION]. The interactions are considered as signals, varying between strong positive (“Click”) to weak positive (“Open”) to weak negative (“No-action”) and strong negative signal (“Complaint” and “Opt-out”).

To allow the model to learn from both weak and strong signals (and since click behaviour is relatively sparse) we assign the interactions with continuous numerical values to account for the signal “magnitude” and positive/negative sign. Continuous values for customer interactions are regarded as hyper-parameters, set as {CLICK: 5, OPEN: 1, NO-ACTION: 0, OPT-OUT: -10, COMPLAINT: -10}. We include email promotion metadata and textual features derived from promotion subjects, texts, and categories.

Model

The model’s input layer is set to encode customer interaction sequences into a sequence embedding. Alongside session interaction sequences, additional model inputs include user-level features (customer metadata such as device, marketing channel and language settings) and sequence-item-level features (e.g. category of item viewed, category of page viewed, or time in seconds since the previous customer interaction). Such features are aggregated using concatenation merge (Mizrachi and Levin, 2019; de Souza Pereira Moreira et al. 2021), in which the embedding of each sequence element is concatenated with the additional input features (as categorical embeddings or continuous values).

Email text metadata, such as promotion categories, promotion names, email subjects and texts, are first merged and then subjected to a text cleaning process. Then, BLOOM encodes the texts into embeddings. BLOOM embeddings enable the system to effectively process multi-lingual email content across different markets.

High-level architecture of the model and feature transformation workflow

The transformer component, built on the XLNet architecture, is designed to model temporal and contextual relationships within the session. The representation of a particular interaction dynamically depends on the temporal position and context, i.e. the values of the remaining interactions in the sequence. The Transformer uses self-attention mechanisms to dynamically adapt the representation of each interaction based on the session’s context.

The model incorporates dual prediction tasks: (1) Next-item prediction task, predicting the next-best-action for a user within a session, and (2) Regression task for predicting continuous value related to user-campaign engagement. The first objective helps the model learn the most relevant call-to-action, and the second objective predicts user engagement per a proposed email marketing campaign. The integration of these tasks into the model’s architecture allows it to simultaneously optimize for accuracy in item recommendations, and predict user-campaign interaction scores.

Model production and scaling

The model is an ensemble that integrates the (1) Merlin NVTabular feature transformation workflow and a (2) Torch-based Transformers4Rec model.

The feature transformation workflow is responsible for processing customer events and features from the back-end system. Within this workflow, customer events within a session sequence are grouped, encoded, and transformed to align with model inputs. This ensures the quality and consistency of events processing, maintains uniform encoding of categorical features between training and inference, and handles the standardization and normalization of continuous features, as well as aggregating sequential data.

Data transformation for training, as well as the model training itself, can be resource-intensive, particularly with large volumes of training data. To address this, we employ a retraining pipeline where feature transformation, model training, and inference are accelerated by GPUs (Tesla A100), available through GCP Vertex AI and running on NVIDIA Merlin PyTorch pre-built containers.

The feature transformation workflow and PyTorch model components are designed for real-time serving at scale (using Triton Inference Server, see figure below). However, in this case we are using them for batch predictions, generating marketing email campaign recommendations for sets of batched customer data.

Training and inference flow for an NVTabular and PyTorch ensemble, deployed with Triton Inference Server (source: *https://nvidia-merlin.github.io/Transformers4Rec/stable/examples/tutorial/04-Inference-with-Triton.html*)

Model Evaluation

The model’s evaluation will be conducted through an A/B test, where a randomly selected group of customers (eligible to receive a marketing email) will be exposed to programs predicted by the model to generate the highest engagement. This group will be compared to a control group, which will receive campaigns determined by the existing rule-based system. Special attention should be given to instances where the campaign predicted by the model is the same as the one selected by the rule-based heuristics, as these cases may introduce noise to the evaluation of the model’s impact (Kohavi et al., 2020, p. 212; Bernardi et al., 2019).

The offline evaluation of the model is conducted on a test set (10% of the sample) containing web and email interactions of customers who were not part of the train set (80%) and the evaluation set (10%), from the same country and retail market. We excluded customer events that occurred after an interaction with the email (and, after a threshold time after the email was sent, where no interaction was observed). By splitting the sample based on customers we hope to limit leakage and spillover effects between events.

Distribution of engagement prediction scores.

The mean and median (scaled) predicted engagement scores per customer-email pairs were higher for emails that were clicked (mean = 0.1609, median = 0.1292) compared to those without a click (mean = 0.1421, median = 0.1215). The distribution of prediction scores for clicked emails skews rightward compared to non-clicked emails, indicating higher frequencies of elevated scores among the clicked group (Mann-Whitney U-test result indicates statistical significance).

Mean “Click”, “Open” and “Optout”-”Complaint” rates for the top 10 recommended campaigns vs. all recommended campaigns. Y-axis values are standardised.

Examining the model’s performance on a test set, for the top 10 ranked campaigns, we observe a significant improvement in click rates, with a 28.63% increase compared to the broader campaign distributions. This indicates that the model is effective in prioritizing emails that optimize click-through rates (CTR). Model results also show a small, statistically insignificant increase in open rates of 2.13%.

Summary

We present a session-based email campaign recommender designed to personalize marketing email communications for IKEA customers. By leveraging both customer interaction data and email text features, we developed a model that combines Transformers4Rec with BLOOM, to identify customer needs and recommend the next-best email campaign for optimal click-through rates.

Offline evaluations indicate improvements in customer interactions, with the top 10 recommended email campaigns. Key learnings suggest the potential to boost email engagement by integrating LLM-generated context embeddings, employing session-based sequence modeling using transformer architecture, and optimizing for multiple customer behavior signals.

References

de Souza Pereira Moreira, G., Rabhi, S., Lee, J. M., Ak, R., & Oldridge, E. (2021, September). Transformers4rec: Bridging the gap between nlp and sequential/session-based recommendation. In Proceedings of the 15th ACM conference on recommender systems (pp. 143–153). https://doi.org/10.1145/3460231.3474255

Bernardi, L., Mavridis, T., & Estevez, P. (2019, July). 150 successful machine learning models: 6 lessons learned at booking. com. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 1743–1751).

Fang, J. (2021). Session-based recommendation with self-attention networks. https://doi.org/10.48550/arXiv.2102.01922

Gao, C., He, X., Gan, D., Chen, X., Feng, F., Li, Y., Chua, T., & Jin, D. (2019). Neural multi-task recommendation from multi-behavior data. In 2019 IEEE 35th International Conference on Data Engineering (ICDE) (pp. 1554–1557). https://doi.org/10.1109/ICDE.2019.00140

Hadash, G., Sar Shalom, O., & Osadchy, R. (2018). Rank and rate: Multi-task learning for recommender systems. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys ‘18). Association for Computing Machinery. https://doi.org/10.1145/3240323.3240406

Hidasi, B., Karatzoglou, A., Baltrunas, L., & Tikk, D. (2016). Session-based recommendations with recurrent neural networks. https://arxiv.org/abs/1511.06939

Kang, W.-C., & McAuley, J. (2018). Self-attentive sequential recommendation. In 2018 IEEE International Conference on Data Mining (ICDM) (pp. 197–206). https://doi.org/10.1109/ICDM.2018.00035

Kohavi, R., Tang, D., & Xu, Y. (2020). *Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press.

Le Scao, T., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al. (2023). Bloom: A 176b-parameter open-access multilingual language model. https://arxiv.org/abs/2301.01220

Liu, Q., Zeng, Y., Mokhosi, R., & Zhang, H. (2018). STAMP: Short-term attention/memory priority model for session-based recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 1831–1839). https://doi.org/10.1145/3219819.3219917

Meng, W., Yang, D., & Xiao, Y. (2020). Incorporating user micro-behaviors and item knowledge into multi-task learning for session-based recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ‘20). Association for Computing Machinery. https://doi.org/10.1145/3397271.3401098

Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. *arXiv*. https://arxiv.org/abs/1310.4546

Mizrachi, M., & Levin, G. (2019). Combining context features in sequence-aware recommender systems. In RecSys ’19 (Late-Breaking Results) (pp. 11–15). https://doi.org/10.1145/3240323.3240406

Ruder, S. (2017). An overview of multi-task learning in deep neural networks. https://arxiv.org/abs/1706.05098

Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., & Jiang, P. (2019). BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. https://arxiv.org/abs/1904.06690

Sun, S., Tang, Y., Dai, Z., & Zhou, F. (2019). Self-attention network for session-based recommendation with streaming data input. IEEE Access, 7, 110499–110509. https://doi.org/10.1109/ACCESS.2019.2931945

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems.https://arxiv.org/abs/1706.03762

Wu, C., Wu, F., Qi, T., & Huang, Y. (2021). Empowering news recommendation with pre-trained language models. https://arxiv.org/abs/2104.07413

Wu, L., Zheng, Z., Qiu, Z., Wang, H., Gu, H., Shen, T., Qin, C., Zhu, C., Zhu, H., Liu, Q., Xiong, H., & Chen, E. (2023). A survey on large language models for recommendation. https://arxiv.org/abs/2305.19860

Xiao, S., Liu, Z., Shao, Y., Di, T., Middha, B., Wu, F., & Xie, X. (2022). Training large-scale news recommenders with pre-trained language models in the loop. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ‘22) (pp. 4215–4225). Association for Computing Machinery. https://doi.org/10.1145/3534678.3539120

Yang, H., Wang, T., Tang, X., Li, Q., Shi, Y., Jiang, S., Yu, H., & Song, H. (2021). Multi-task learning for bias-free joint CTR prediction and market price modeling in online advertising. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (CIKM ‘21) (pp. 2291–2300). Association for Computing Machinery. https://doi.org/10.1145/3459637.3482373

Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2020). XLNet: Generalized autoregressive pretraining for language understanding. https://arxiv.org/abs/1906.08237

Zhao, Z., Fan, W., Li, J., Liu, Y., Mei, X., Wang, Y., Wen, Z., Wang, F., Zhao, X., Tang, J., & Li, Q. (2024). Recommender systems in the era of large language models (LLMs). https://arxiv.org/abs/2307.02046

Elevating IKEA’s Email Marketing Strategy: Assembling attention in next-best email campaigns was originally published in Flat Pack Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Newsfeed

News from Ingka Digital

Ingka's Flat Pack Tech at Medium.com