{"id":1185,"date":"2024-05-18T16:17:07","date_gmt":"2024-05-18T16:17:07","guid":{"rendered":"https:\/\/ml-gis-service.com\/?p=1185"},"modified":"2024-05-18T16:25:56","modified_gmt":"2024-05-18T16:25:56","slug":"wsknn-session-based-recommendation-engine-and-its-parameters","status":"publish","type":"post","link":"https:\/\/ml-gis-service.com\/index.php\/2024\/05\/18\/wsknn-session-based-recommendation-engine-and-its-parameters\/","title":{"rendered":"Set this right! &#8211; How to prepare recommendation system for the real world"},"content":{"rendered":"\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p>Session-based recommendation engine in Python<\/p>\n<cite>Part 4<\/cite><\/blockquote>\n\n\n\n<h1 class=\"wp-block-heading\">Introduction<\/h1>\n\n\n\n<p>Every model is founded on high-quality data. Assuming a clean and representative dataset, you may start thinking about the model&#8217;s hyperparameters. <code>WSKNN<\/code> recommendation model has multiple settings, and we will test them to see how they affect recommendations. Before we start, you should know that if you use <code>WSKNN<\/code> package and its internal algorithm, you can change model parameters on the fly, even during the inference! What&#8217;s the catch? The model grows very fast because it duplicates data; thus, it is unreliable for enormous datasets.<\/p>\n\n\n\n<p>Here is the list of model parameters in the package <code>wsknn==1.2<\/code>:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>The number of recommendations.<\/li>\n\n\n\n<li>Sample size.<\/li>\n\n\n\n<li>Sampling strategy.<\/li>\n\n\n\n<li>Number of neighbors.<\/li>\n\n\n\n<li>Weighting function.<\/li>\n\n\n\n<li>Ranking strategy.<\/li>\n\n\n\n<li>Known items.<\/li>\n\n\n\n<li>Required action and its index.<\/li>\n\n\n\n<li>Custom weights.<\/li>\n\n\n\n<li>Random recommendations.<\/li>\n<\/ol>\n\n\n\n<p>We will describe each parameter and check how single or multiple parameters can change the model scoring and inference time.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">(1) Number of recommendations<\/h2>\n\n\n\n<p><strong>Parameter name:<\/strong>&nbsp;<code>number_of_recommendations<\/code><\/p>\n\n\n\n<p><strong>Default value:<\/strong>&nbsp;5<\/p>\n\n\n\n<p><strong>Short description:<\/strong>&nbsp;the number of recommended products<\/p>\n\n\n\n<p><strong>Long description:<\/strong>\u00a0This parameter doesn&#8217;t switch anything within a trained model but affects the number of outputs. We show five recommendations for online shops typically. Sometimes, we need more results. <code>WSKNN<\/code> returns every recommendation with a weight; thus, we can set this parameter to a large value and cut off low-weighted recommendations. UI\/UX design might impose how many recommendations a model returns. This parameter is related to (7) and (10) &#8211; those three parameters profoundly affect the output.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">(2) Sample size<\/h2>\n\n\n\n<p><strong>Parameter name:<\/strong>&nbsp;<code>sample_size<\/code><\/p>\n\n\n\n<p><strong>Default value:<\/strong>&nbsp;1000<\/p>\n\n\n\n<p><strong>Short description:<\/strong>&nbsp;how many sessions from the model are sampled to make a recommendation, the possible neighbors space<\/p>\n\n\n\n<p><strong>Long description:<\/strong>&nbsp;This is the first parameter that changes the internal workings of the WSKNN architecture. Typically, the model stores billions of sessions. Checking every session&#8217;s similarity could be cumbersome. The model makes a recommendation using only a subset of all available sessions. It chooses the subset using the strategy provided by the parameter (3) <strong>Sampling Strategy<\/strong>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">(3) Sampling strategy<\/h2>\n\n\n\n<p><strong>Parameter name:<\/strong>&nbsp;<code>sampling_strategy<\/code><\/p>\n\n\n\n<p><strong>Default value:<\/strong>&nbsp;<code>\"common_items\"<\/code><\/p>\n\n\n\n<p><strong>Short description:<\/strong>&nbsp;initial session-filtering options<\/p>\n\n\n\n<p><strong>Long description:<\/strong>&nbsp;The WSKNN network has thousands of sessions, and it would be unreliable to check the similarity with every session in a set. That&#8217;s why we set the <strong>sample size<\/strong> (2) and chose a sampling strategy from four options:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>\"random\"<\/code>: algorithm picks a random subset of sessions. It prevents overfitting and works nicely with the <strong>random recommendations parameter<\/strong> (10)<\/li>\n\n\n\n<li><code>\"recent\"<\/code>: algorithm selects the most recent records. Invaluable when data presents cyclic or seasonal patterns, short-lived trends, and anomalies<\/li>\n\n\n\n<li><code>\"common_items\"<\/code>: we want to recommend products that are most frequently grouped within a session<\/li>\n\n\n\n<li><code>\"weighted_events\"<\/code>: we can assign weights to each event; for example, actions like product view, go to checkout, and purchase will have different weights. We use sessions with the highest weights (usually those ending with purchase)<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">(4) Number of neighbors<\/h2>\n\n\n\n<p><strong>Parameter name:<\/strong>&nbsp;<code>number_of_neighbors<\/code><\/p>\n\n\n\n<p><strong>Default value:<\/strong>&nbsp;10<\/p>\n\n\n\n<p><strong>Short description:<\/strong>&nbsp;the number of the closest sessions to choose the items from<\/p>\n\n\n\n<p><strong>Long description:<\/strong>&nbsp;The set of <strong>possible neighbors<\/strong> (2, 3) is limited again after <strong>session-weighting<\/strong> (5) of the possible neighbors. Why? Because the item space could still be too large to make fast calculations. From these neighbors, we will rank items and make recommendations. The figure below shows the whole process and links parameters to each process step.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"576\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/algorithm-internals-1024x576.png\" alt=\"\" class=\"wp-image-1211\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/algorithm-internals-1024x576.png 1024w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/algorithm-internals-300x169.png 300w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/algorithm-internals-768x432.png 768w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/algorithm-internals-1536x864.png 1536w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/algorithm-internals.png 1680w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<h2 class=\"wp-block-heading\">(5) Weighting function<\/h2>\n\n\n\n<p><strong>Parameter name:<\/strong>&nbsp;<code>weighting_func<\/code><\/p>\n\n\n\n<p><strong>Default value:<\/strong>&nbsp;<code>\"linear\"<\/code><\/p>\n\n\n\n<p><strong>Short description:<\/strong>&nbsp;The possible neighbor&#8217;s sessions weighting and ranking method<\/p>\n\n\n\n<p><strong>Long description:<\/strong>&nbsp;Using this parameter, we rank the <strong>possible neighbors<\/strong> from steps (2, 3) and create the <strong>closest neighbors<\/strong> set of the size given in step (4). We have three possible weighting functions, and with every function, we compare the user session for recommendation to the possible neighbors&#8217; sessions. Items ordering is essential!<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>\"linear\"<\/code>: weight is proportional to the position of an item in a possible neighbor session. The newest elements (last on the list) get higher weights.<\/li>\n\n\n\n<li><code>\"log\"<\/code>: works like <code>\"linear\"<\/code> but older elements get smaller weights. This function mimics short-term memory, emphasizes the newest elements, and gives larger weights for short-sequence items.&nbsp;<\/li>\n\n\n\n<li><code>\"quadratic\":<\/code> similarly to <code>\"log\"<\/code> penalizes the oldest elements in sequence more.<\/li>\n<\/ul>\n\n\n\n<p>We get ranked sessions of the possible neighbors, then sort them and slice them up to the number of the closest neighbors. Those sessions go to the next step of recommendation, where we <strong>rank items<\/strong> (products) (6).<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">(6) Ranking Strategy<\/h2>\n\n\n\n<p><strong>Parameter name:<\/strong>&nbsp;<code>ranking_strategy<\/code><\/p>\n\n\n\n<p><strong>Default value:<\/strong>&nbsp;<code>\"linear\"<\/code><\/p>\n\n\n\n<p><strong>Short description:<\/strong>&nbsp;possible recommendations ranking method<\/p>\n\n\n\n<p><strong>Long description:<\/strong>&nbsp;Having a subset of the closest neighbors, we rank all items that have occurred in the nearest neighbors&#8217; sessions. The process is similar to the session weighting. Each weight is averaged (it is calculated for each item in each closest neighbor session):<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>\"linear\"<\/code>: weight is proportional to the position of the item in session; if the item index is -10 or less, then it gets a 0 score.<\/li>\n\n\n\n<li><code>\"inv\"<\/code>: a simple score where item weight is calculated as an inverted item index.<\/li>\n\n\n\n<li><code>\"log\"<\/code>: works like <code>\"linear\"<\/code> but older elements get smaller weights. This function mimics short-term memory, emphasizes the newest elements, and gives larger weights for short-sequence items.&nbsp;<\/li>\n\n\n\n<li><code>\"quadratic\"<\/code>: similarly to <code>\"log\"<\/code> penalizes the oldest item indexes in sequence more.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">(7) Known Items<\/h2>\n\n\n\n<p><strong>Parameter name:<\/strong>&nbsp;<code>return_events_from_session<\/code><\/p>\n\n\n\n<p><strong>Default value:<\/strong>&nbsp;<code>True<\/code><\/p>\n\n\n\n<p><strong>Short description:<\/strong>&nbsp;should the recommender return items with which the user has had interaction?<\/p>\n\n\n\n<p><strong>Long description:<\/strong>&nbsp;Business purposes drive this parameter. We must decide if we want our customers to learn about the new products (then we set this parameter to <code>False<\/code>) or to speed up the purchase and order, showing items known to the user. Using known items will set scoring metrics higher BUT can harm the user experience and monetization. This parameter could be the first to tweak in production mode.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">(8) Required action and its index<\/h2>\n\n\n\n<p><strong>Parameter name:<\/strong>&nbsp;<code>required_sampling_event<\/code> and <code>required_sampling_event_index<\/code><\/p>\n\n\n\n<p><strong>Default value:<\/strong>&nbsp;<code>None<\/code> and <code>None<\/code><\/p>\n\n\n\n<p><strong>Short description:<\/strong>&nbsp;Additional condition (action, event type) required for a session to land in the possible neighbors&#8217; subset<\/p>\n\n\n\n<p><strong>Long description:<\/strong>&nbsp;Sometimes, we expect a particular action from the users. For example, it could be a purchase. In this scenario, we could experiment with the recommendation space and limit it only to the sessions with the purchase event. We can filter those sessions BEFORE the model fitting. Still, the model allows us to test recommendations AFTER fitting with the requirement of an additional row with actions\/events linked to each item in a session. We must pass this row index (because there is a possibility that we pass <strong>custom weights<\/strong> (9), and this could be the 2nd or 3rd row in a session).<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">(9) Custom weights<\/h2>\n\n\n\n<p><strong>Parameter name:<\/strong>&nbsp;<code>sampling_event_weights_index<\/code><\/p>\n\n\n\n<p><strong>Default value:<\/strong>&nbsp;<code>None<\/code><\/p>\n\n\n\n<p><strong>Short description:<\/strong>&nbsp;Index of the row with custom item (event) weights<\/p>\n\n\n\n<p><strong>Long description:<\/strong>&nbsp;We can pass an additional row in each session with custom weights applied to each event. Then, it could be used for the possible neighbor selection when we set the <strong>sampling strategy<\/strong> (3) to <code>\"weighted_events\"<\/code>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">(10) Random recommendations<\/h2>\n\n\n\n<p><strong>Parameter name:<\/strong>&nbsp;<code>recommend_any<\/code><\/p>\n\n\n\n<p><strong>Default value:<\/strong>&nbsp;<code>False<\/code><\/p>\n\n\n\n<p><strong>Short description:<\/strong>&nbsp;should the recommender always return the maximum number of recommendations?<\/p>\n\n\n\n<p><strong>Long description:<\/strong>&nbsp;Sometimes, the recommender won&#8217;t find items for recommendation. Out of 10, it will pick only 3. This parameter tells the model what to do in this case. In some circumstances, the middleware is responsible for filling the empty slots. But in other cases, the recommender should always return fixed-length output, and then setting this parameter to <code>True<\/code> is a good idea.<\/p>\n\n\n\n<h1 class=\"wp-block-heading\">Experiments<\/h1>\n\n\n\n<p>We will learn how those parameters affect experiments&#8217; theoretical scoring and practical inference time. There could be much more, and we discuss other cases in the following article about business strategy when we run a recommendation engine. For now, we will focus on analytical and engineering perspectives. Experiments can be viewed in the <a href=\"https:\/\/github.com\/SimonMolinsky\/blog-post-wsknn-movielens\/tree\/main\/part4\">GitHub Repository HERE<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Setup<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Step 1<\/h3>\n\n\n\n<p>Download the MovieLens dataset (MovieLens 100k). You can get data from the tutorial\u2019s repository here: [1].<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 2<\/h3>\n\n\n\n<p>Create <code>mamba<\/code> environment or virtual environment.<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"bash\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">mamba create -n wsknn Python=\u201d3.11\u201d<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3<\/h3>\n\n\n\n<p>Activate the environment, install <code>pip<\/code>, <code>seaborn<\/code>, <code>matplotlib<\/code>, <code>pandas<\/code> and <code>notebook<\/code> from <code>mamba<\/code>, and then install <code>wsknn<\/code> from <code>pip<\/code>.<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"bash\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">mamba activate wsknn\n(wsknn) mamba install pip notebook seaborn matplotlib pandas numpy tqdm\n(wsknn) pip install wsknn<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 4<\/h3>\n\n\n\n<p>Open Jupyter Notebook and create a new Python3 notebook.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 5 Import packages, load data, prepare model<\/h3>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\"># Imports\nfrom typing import Dict, List, Union\nfrom datetime import datetime\n\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nfrom wsknn import fit, predict\nfrom wsknn.evaluate import score_model\nfrom wsknn.preprocessing.parse_static import parse_flat_file\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n\n# Functions and classes\ndef generate_parameter_set(number_of_recommendations: int = 5,\n                           number_of_neighbors: int = 10,\n                           sampling_strategy: str = 'common_items',\n                           sample_size: int = 1000,\n                           weighting_func: str = 'linear',\n                           ranking_strategy: str = 'linear',\n                           return_events_from_session: bool = True,\n                           required_sampling_event: Union[int, str] = None,\n                           required_sampling_event_index: int = None,\n                           sampling_str_event_weights_index: int = None,\n                           recommend_any: bool = False) -> Dict:\n    \"\"\"\n    Function generates multiple parameter sets.\n    \"\"\"\n    d = {\n        'number_of_recommendations': number_of_recommendations,\n        'number_of_neighbors': number_of_neighbors,\n        'sampling_strategy': sampling_strategy,\n        'sample_size': sample_size,\n        'weighting_func': weighting_func,\n        'ranking_strategy': ranking_strategy,\n        'return_events_from_session': return_events_from_session,\n        'recommend_any': recommend_any,\n        'required_sampling_event': required_sampling_event,\n        'required_sampling_event_index': required_sampling_event_index,\n        'sampling_str_event_weights_index': sampling_str_event_weights_index\n    }\n    return d\n\n\ndef plot_scores_barplot(dataset, category_col, score_type):\n    plt.figure(figsize=(8, 5))\n    sns.barplot(dataset, x=category_col, y=score_type)\n    plt.show()\n\n\ndef plot_scores_heatmap(dataset, rows, cols, values):\n    _pivoted = dataset.pivot(index=rows, columns=cols, values=values)\n    plt.figure(figsize=(5, 5))\n    sns.heatmap(_pivoted, cmap='viridis')\n    plt.show()\n\n\ndef train_validate_samples(set_of_sessions):\n    \n    sessions_keys = list(set_of_sessions.keys())\n    n_sessions = int(0.1 * len(sessions_keys))\n    key_sample = np.random.choice(sessions_keys, n_sessions)\n    \n    training_set = {_key: set_of_sessions[_key] for _key in sessions_keys if _key not in key_sample}\n    validation_set = [set_of_sessions[_key] for _key in key_sample]\n    \n    return training_set, validation_set\n\n\n# Class which stores all model's and their results\nclass TestModels:\n\n    def __init__(self, training_set: Dict, test_set: List, psets: List):\n        self.training_set = training_set\n        self.test_set = test_set\n        self.psets = psets\n        self.scoring_results = self.get_scoring()\n\n    def get_scoring(self):\n        \"\"\"\n        Method scores multiple different models\n        \"\"\"\n        scorings = []\n        for params in tqdm(self.psets):\n            model = fit(sessions=self.training_set, **params)\n            scores = score_model(sessions=self.test_set, trained_model=model, k=5)\n            scores.update(params)\n            scorings.append(scores)\n\n        scoring_results = pd.DataFrame(scorings)\n        return scoring_results\n\n    def scores(self):\n        return self.scoring_results\n\n\nclass TestModelResponseTime:\n\n    def __init__(self, training_set: Dict, test_set: List, psets: List):\n        self.training_set = training_set\n        self.test_set = test_set\n        self.psets = psets\n        self.time_measurement = self.get_time()\n\n    def get_time(self):\n        \"\"\"\n        Method calculates recommendation times for each set of parameters\n        \"\"\"\n\n        results = []\n        \n        for params in tqdm(self.psets):\n            model = fit(sessions=self.training_set, **params)\n            t0 = datetime.now()\n            _ = [\n                predict(model, list(_s)) for _s in self.test_set\n            ]\n            tx = (datetime.now() - t0).total_seconds()\n            d = {}\n            d['dt-seconds'] = tx\n            d.update(params)\n            results.append(d)\n\n        measured_results = pd.DataFrame(results)\n        return measured_results\n\n    def measurements(self):\n        return self.time_measurement\n\n\n# Load data\nfpath = 'ml-100k\/u.data'\n# As action we assume rating\nallowed_actions = {\n    '1': 1,\n    '2': 2,\n    '3': 3,\n    '4': 4,\n    '5': 5\n}\nds = parse_flat_file(fpath,\n                     sep='\\t',\n                     session_index=0,\n                     product_index=1,\n                     time_index=3,\n                     action_index=2,\n                     allowed_actions=allowed_actions,\n                     time_to_numeric=True)\n\ntraining_ds, validation_ds = train_validate_samples(ds[1].session_items_actions_map)<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Experiment 1: Scoring vs Sample Size and Sampling Strategy<\/h2>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\"># Setting parameters\npossible_neighbors_sizes = [100, 200, 500, 1000]\npossible_neighbors_sampling_strategies = [\"random\", \"common_items\", \"recent\", \"weighted_events\"]\nsampling_str_event_weights_index = -1\nnumber_of_recommendations = 5\n\nexperiment_1_parameters = []\n\nfor possible_n_size in possible_neighbors_sizes:\n    for samp_strat in possible_neighbors_sampling_strategies:\n        experiment_1_parameters.append(\n            generate_parameter_set(\n                number_of_recommendations=number_of_recommendations,\n                sample_size=possible_n_size,\n                sampling_strategy=samp_strat,\n                sampling_str_event_weights_index=sampling_str_event_weights_index\n            )\n        )\n\n# Score models\nexp1_test = TestModels(\n    training_set=training_ds,\n    test_set=validation_ds,\n    psets=experiment_1_parameters\n)\n\ndf = exp1_test.scores()<\/pre>\n\n\n\n<p>In the first experiment, we compare sampling strategies and Mean Reciprocal Rank, Precision, and Recall scores. (If you want to know more about those metrics, check the second article in the series.) Colorful plots tell better stories than tables with numbers, so let&#8217;s plot every metric as a heatmap with two dimensions: one representing sample size and the other sampling strategy.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">MRR<\/h3>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">plot_scores_heatmap(df, 'sample_size', 'sampling_strategy', 'MRR')<\/pre>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"461\" height=\"552\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp1-mrr.png\" alt=\"\" class=\"wp-image-1190\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp1-mrr.png 461w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp1-mrr-251x300.png 251w\" sizes=\"auto, (max-width: 461px) 100vw, 461px\" \/><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\">Precision<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"452\" height=\"552\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp1-prec.png\" alt=\"\" class=\"wp-image-1191\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp1-prec.png 452w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp1-prec-246x300.png 246w\" sizes=\"auto, (max-width: 452px) 100vw, 452px\" \/><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\">Recall<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"461\" height=\"552\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp1-rec.png\" alt=\"\" class=\"wp-image-1192\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp1-rec.png 461w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp1-rec-251x300.png 251w\" sizes=\"auto, (max-width: 461px) 100vw, 461px\" \/><\/figure><\/div>\n\n\n<p>Depending on your use case, you should maximize Precision or Recall. Probably <code>recent<\/code> sampling strategy is the best, it works well for every sample size.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Experiment 2: Scoring vs Sample Size &amp; Number of Neighbors<\/h2>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">possible_neighbors_sizes = [250, 500, 1000, 2000]\nclosest_neighbors_sizes = [10, 50, 100, 250]\npossible_neighbors_sampling_strategy = \"recent\"\nnumber_of_recommendations = 5\n\nexperiment_2_parameters = []\n\nfor possible_n_size in possible_neighbors_sizes:\n    for closest_n_size in closest_neighbors_sizes:\n        experiment_2_parameters.append(\n            generate_parameter_set(\n                number_of_recommendations=number_of_recommendations,\n                sample_size=possible_n_size,\n                sampling_strategy=possible_neighbors_sampling_strategy,\n                number_of_neighbors=closest_n_size\n            )\n        )\n\nexp2_test = TestModels(\n    training_set=training_ds,\n    test_set=validation_ds,\n    psets=experiment_2_parameters\n)\n\ndf = exp2_test.scores()\n\nplot_scores_heatmap(df, 'sample_size', 'number_of_neighbors', 'MRR')\nplot_scores_heatmap(df, 'sample_size', 'number_of_neighbors', 'Recall')\nplot_scores_heatmap(df, 'sample_size', 'number_of_neighbors', 'Precision')<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">MRR<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"452\" height=\"448\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp2-scores.png\" alt=\"\" class=\"wp-image-1194\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp2-scores.png 452w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp2-scores-300x297.png 300w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp2-scores-150x150.png 150w\" sizes=\"auto, (max-width: 452px) 100vw, 452px\" \/><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\">Recall<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"461\" height=\"448\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp2-recall-1.png\" alt=\"\" class=\"wp-image-1197\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp2-recall-1.png 461w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp2-recall-1-300x292.png 300w\" sizes=\"auto, (max-width: 461px) 100vw, 461px\" \/><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\">Precision<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"452\" height=\"448\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp2-prec.png\" alt=\"\" class=\"wp-image-1196\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp2-prec.png 452w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp2-prec-300x297.png 300w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp2-prec-150x150.png 150w\" sizes=\"auto, (max-width: 452px) 100vw, 452px\" \/><\/figure><\/div>\n\n\n<p>We clearly see that when the model picks the closest neighbors, the scoring is better.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Experiment 3: Scoring&nbsp;vs&nbsp;Weighting Function<\/h2>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">possible_neighbors_size = 1000\nclosest_neighbors_size = 500\nneighbors_sampling_strategy = \"recent\"\nnumber_of_recommendations = 5\nweighting_funcs = ['linear', 'log', 'quadratic']\n\nexperiment_3_parameters = [\n    generate_parameter_set(\n        number_of_recommendations=number_of_recommendations,\n        sample_size=possible_neighbors_size,\n        sampling_strategy=neighbors_sampling_strategy,\n        number_of_neighbors=closest_neighbors_size,\n        weighting_func=wfunc) for wfunc in weighting_funcs\n]\n\nexp3_test = TestModels(\n    training_set=training_ds,\n    test_set=validation_ds,\n    psets=experiment_3_parameters\n)\n\ndf = exp3_test.scores()\n\nplot_scores_barplot(df, 'weighting_func', 'MRR')\nplot_scores_barplot(df, 'weighting_func', 'Recall')\nplot_scores_barplot(df, 'weighting_func', 'Precision')<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">MRR<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"691\" height=\"294\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp3-mrr.png\" alt=\"\" class=\"wp-image-1199\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp3-mrr.png 691w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp3-mrr-300x128.png 300w\" sizes=\"auto, (max-width: 691px) 100vw, 691px\" \/><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\">Recall<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"294\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp3-recall.png\" alt=\"\" class=\"wp-image-1200\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp3-recall.png 700w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp3-recall-300x126.png 300w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\">Precision<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"691\" height=\"294\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp3-prec.png\" alt=\"\" class=\"wp-image-1201\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp3-prec.png 691w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp3-prec-300x128.png 300w\" sizes=\"auto, (max-width: 691px) 100vw, 691px\" \/><\/figure><\/div>\n\n\n<p>The differences between the session-weighting functions seem to be very small for this realization.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Experiment 4: Scoring vs Ranking Strategy<\/h2>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">possible_neighbors_size = 1000\nclosest_neighbors_size = 500\nneighbors_sampling_strategy = \"recent\"\nnumber_of_recommendations = 5\nweighting_func = 'log'\nranking_strategies = ['linear', 'inv', 'quadratic', 'log']\n\nexperiment_4_parameters = [\n    generate_parameter_set(\n        number_of_recommendations=number_of_recommendations,\n        sample_size=possible_neighbors_size,\n        sampling_strategy=neighbors_sampling_strategy,\n        number_of_neighbors=closest_neighbors_size,\n        weighting_func=weighting_func,\n        ranking_strategy=r_str\n    ) for r_str in ranking_strategies\n]\n\nexp4_test = TestModels(\n    training_set=training_ds,\n    test_set=validation_ds,\n    psets=experiment_4_parameters\n)\n\ndf = exp4_test.scores()\n\nplot_scores_barplot(df, 'ranking_strategy', 'MRR')\nplot_scores_barplot(df, 'ranking_strategy', 'Recall')\nplot_scores_barplot(df, 'ranking_strategy', 'Precision')<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">MRR<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"691\" height=\"298\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp4-mrr.png\" alt=\"\" class=\"wp-image-1204\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp4-mrr.png 691w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp4-mrr-300x129.png 300w\" sizes=\"auto, (max-width: 691px) 100vw, 691px\" \/><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\">Recall<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"294\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp4-rec.png\" alt=\"\" class=\"wp-image-1205\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp4-rec.png 700w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp4-rec-300x126.png 300w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\">Precision<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"691\" height=\"294\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp4-pre.png\" alt=\"\" class=\"wp-image-1206\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp4-pre.png 691w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp4-pre-300x128.png 300w\" sizes=\"auto, (max-width: 691px) 100vw, 691px\" \/><\/figure><\/div>\n\n\n<p>Items-weighting brings much clearer differences between weighting strategies and the model scores. We can assume that <code>log<\/code> strategy will be the best in production.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Experiment 5: Response Time vs Sample Size<\/h2>\n\n\n\n<p>The previous four experiments tested analytical scores. The model is always more than its evaluation metrics; data scientists should check other model parameters. The core engineering parameter is model response time because it might greatly limit the usefulness of the solution in a real-world setting. We will check how the model behaves when we change the size of possible neighbors&#8217; space and sampling strategy.<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">sample_sizes = np.arange(1000, 10001, 200)\nclosest_neighbors = 250\n\nexperiment_5_parameters = []\n\nfor possible_n_size in sample_sizes:\n    experiment_5_parameters.append(\n        generate_parameter_set(\n            sample_size=possible_n_size,\n            number_of_neighbors=closest_neighbors\n        )\n    )\n\nexp5_test = TestModelResponseTime(\n    training_set=training_ds,\n    test_set=validation_ds,\n    psets=experiment_5_parameters\n)\n\ndf = exp5_test.measurements()\n\nplt.figure(figsize=(12, 6))\nplt.plot(df['sample_size'], df['dt-seconds'])\nplt.show()<\/pre>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"990\" height=\"505\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp5-res-time.png\" alt=\"\" class=\"wp-image-1207\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp5-res-time.png 990w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp5-res-time-300x153.png 300w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp5-res-time-768x392.png 768w\" sizes=\"auto, (max-width: 990px) 100vw, 990px\" \/><\/figure><\/div>\n\n\n<p>As you see, the possible neighbors&#8217; size does not affect the response time. To be sure, you should run this experiment multiple times and check the median response time.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Experiment 6: Response Time vs Weighted &amp; Non-weighted Items<\/h2>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">sample_sizes = np.arange(1000, 10001, 500)\nclosest_neighbors = 250\nmethods = ['common_items', 'weighted_events', 'recent', 'random']\nweight_idx = -1\n\nexperiment_6_parameters = []\n\nfor possible_n_size in sample_sizes:\n    for sampling_method in methods:\n        experiment_6_parameters.append(\n            generate_parameter_set(\n                sample_size=possible_n_size,\n                number_of_neighbors=closest_neighbors,\n                sampling_strategy=sampling_method,\n                sampling_str_event_weights_index=weight_idx\n            )\n        )\n\nexp6_test = TestModelResponseTime(\n    training_set=training_ds,\n    test_set=validation_ds,\n    psets=experiment_6_parameters\n)\n\ndf = exp6_test.measurements()\n\nplt.figure(figsize=(12, 6))\nplt.plot(df[df['sampling_strategy'] == 'common_items']['sample_size'], df[df['sampling_strategy'] == 'common_items']['dt-seconds'])\nplt.plot(df[df['sampling_strategy'] == 'random']['sample_size'], df[df['sampling_strategy'] == 'random']['dt-seconds'])\nplt.plot(df[df['sampling_strategy'] == 'recent']['sample_size'], df[df['sampling_strategy'] == 'recent']['dt-seconds'])\nplt.plot(df[df['sampling_strategy'] == 'weighted_events']['sample_size'], df[df['sampling_strategy'] == 'weighted_events']['dt-seconds'])\nplt.legend(['common_items', 'random', 'recent', 'weighted_events'])\nplt.show()<\/pre>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"977\" height=\"505\" src=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp6.png\" alt=\"\" class=\"wp-image-1208\" srcset=\"https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp6.png 977w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp6-300x155.png 300w, https:\/\/ml-gis-service.com\/wp-content\/uploads\/2024\/05\/exp6-768x397.png 768w\" sizes=\"auto, (max-width: 977px) 100vw, 977px\" \/><\/figure><\/div>\n\n\n<p>As you see, the fastest responses are generated for the <code>random<\/code> selection of possible sessions. Here, we don&#8217;t have any surprises.<\/p>\n\n\n\n<h1 class=\"wp-block-heading\">Summary<\/h1>\n\n\n\n<p>Thank you for your time with the article; hopefully, you have enough knowledge to use the WSKNN system in your settings. In the next article, last from the series, we will talk more about business problems with recommendations and how to measure the business performance of our models. We will focus more on some parameters from the group because they can significantly influence the outcomes.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>How to setup recommendation system to get the best results<\/p>\n","protected":false},"author":1,"featured_media":1212,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[18,251,19,3,49,249,31,250],"tags":[7,257,256,252,254],"class_list":["post-1185","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-science","category-e-commerce","category-machine-learning","category-python","category-rd","category-recommendation-engine","category-tutorials","category-wsknn","tag-python","tag-recommender-engine","tag-recommender-system","tag-session-based-recommendations","tag-wsknn"],"_links":{"self":[{"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/posts\/1185","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/comments?post=1185"}],"version-history":[{"count":20,"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/posts\/1185\/revisions"}],"predecessor-version":[{"id":1227,"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/posts\/1185\/revisions\/1227"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/media\/1212"}],"wp:attachment":[{"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/media?parent=1185"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/categories?post=1185"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ml-gis-service.com\/index.php\/wp-json\/wp\/v2\/tags?post=1185"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}