Earlier than constructing our Machine Studying mannequin, supply guarantees have been solely supplied by sellers. As ManoMano operates as a market, these sellers are valued companions to us and play an important function in making certain buyer satisfaction. Nevertheless, to keep away from being penalized for delays, sellers are inclined to overestimate delivery occasions. This conservative strategy comes at a price: their estimations continuously lack accuracy, resulting in inefficiencies and diminishing the client expertise. Regardless of their conservative guarantees, many orders are nonetheless delivered late, negatively impacting the Internet Promoter Rating (NPS).
We observed that over 40% of orders arrived sooner than promised, revealing a chance to optimize and shorten supply occasions with out compromising reliability. Our objective at ManoMano is to supply correct, aggressive, and dependable supply home windows whereas protecting the general late price under 5%. To realize this, our information science group has developed progressively superior machine studying fashions that construct on each other, every addressing particular limitations and introducing new strategies to boost supply time predictions.
Key Metrics
To guage and optimize supply guarantees successfully, we depend on 2 key metrics that replicate buyer expertise and promise reliability. These metrics assist us perceive the stability between offering aggressive supply estimates and sustaining excessive reliability.
- Late Fee: The share of orders the place the precise supply time exceeds the promised most supply time. Preserving the late price low is crucial for enhancing the Internet Promoter Rating (NPS) and making certain a constructive post-purchase expertise.
- Common Most Supply Time (Avg Max): The common higher certain of the supply window promised to prospects. This metric is important for driving conversion charges (CVR), as shorter supply guarantees make merchandise extra interesting to prospects.
Shortening Avg Max can result in a CVR uplift, as prospects usually tend to buy merchandise with aggressive delivery occasions. Nevertheless, lowering Avg Max additionally will increase the danger of late deliveries, which may hurt buyer satisfaction and decrease the NPS. Conversely, extending the supply window reduces the late price however makes the product much less aggressive within the market.
You’ll have observed that we targeted on the utmost supply promise and never the minimal. This selection has been made consciously as a result of insights confirmed us that contemplating a supply window, the principle driver of conversion is the max (not the min nor the window size), and the principle driver of buyer dissatisfaction is the late price (and never the early price).
As illustrated within the determine, Avg Max and Late Fee are negatively correlated. For a given mannequin, compressing Avg Max inevitably will increase the Late Fee. The decrease the curve representing this tradeoff, the higher the mannequin, because it demonstrates improved efficiency in balancing shorter supply guarantees with decrease Late Charges. The objective is to refine the mannequin to attain the optimum stability and push the curve downward.
Mannequin 1: Primary Mannequin Leveraging Historic Information
We developed the primary base mannequin, a historic quantile (higher certain to achieve lower than 5% late price) on the vendor x provider stage. We face two main challenges whereas growing: incomplete information and the issue of estimating quantiles from finite datasets.
Supply time predictions depend on information from accomplished orders, which inherently excludes undelivered or considerably delayed orders. Due to this fact, we constructed a dataset with a 14-day buffer, making certain that 99% of our orders have been included for evaluation.
Moreover, quantile prediction on finite numbers requires cautious calibration. There’s nearly no excellent worth within the real-world dataset that precisely 5% of orders are delayed. As proven within the instance of 1 vendor within the determine under, when aiming for a supply window of three days, the late price was noticed at 1%. Decreasing the window to 2 days induced the late price to leap to 18%, underscoring the sensitivity of predictions to minor changes.
To mitigate this, we first calculated the 5% historic quantile on the vendor × provider stage, representing a conservative estimate of the utmost supply time. To optimize additional, we launched flexibility by barely adjusting the late price limits on the vendor stage. This strategy allowed us to distribute optimistic and pessimistic estimates throughout totally different sellers whereas making certain the general late price remained under 5%.
This dynamic adjustment was significantly efficient in addressing variability in delivery time distributions. As an example, as illustrated within the determine under, the supply time distribution for Vendor 1 is tightly concentrated. A 1-day discount in Avg Max right here leads to a 5% improve within the late price. Conversely, Vendor 2’s supply time distribution is way broader. For this vendor, lowering the Avg Max from 7 days to 2 days impacts solely the 5% late price threshold whereas attaining a big discount of 5 days in Avg Max.
This base mannequin successfully addressed the constraints of sparse and censored information, supplied precious insights into seller-carrier dynamics, and established a baseline for additional enhancements. By way of efficiency, whereas barely growing the late price, the mannequin efficiently saved it below the 5% threshold and lowered the Avg Max by 0.5 days.
Mannequin 2: Logarithmic Transformation and Prediction Intervals with Linear Regression
Whereas the primary mannequin supplied a foundational framework for optimizing supply guarantees, it processed information in isolation, limiting its skill to leverage broader patterns. To beat this limitation, we developed a second mannequin utilizing a machine learning-based strategy to raised combine historic information throughout totally different granularities.
Not like frequent prediction fashions, which goal for accuracy round a central imply (y_predict = y_truth), we want a mannequin that focuses on prediction intervals. This mannequin targets the 95% percentile of supply occasions because the higher certain and the 5% percentile because the decrease certain. By doing so, the mannequin ensures that y_predict_upper >= 95% y_truth and y_predict_lower ≤ 95% y_truth, offering dependable supply home windows with well-calibrated confidence intervals.
One of many key challenges addressed by this mannequin is the right-skewed distribution of supply occasions in our dataset. Normal linear regression fashions, which depend on strange least squares (OLS) assumptions, are designed for information that follows a traditional distribution. Making use of OLS on to right-skewed information introduces prediction bias, undermining the reliability of the supply promise.
To mitigate this situation, we utilized a logarithmic transformation to the uncooked supply time information, reshaping the distribution to approximate normality. With this remodeled information, we applied a linear regression mannequin utilizing statsmodels, which allowed us to calculate the 95% confidence interval because the higher certain and the 5% confidence interval because the decrease certain of the supply window.
The second mannequin demonstrated notable enhancements in efficiency over the primary. Whereas sustaining the identical late price, it additional lowered the typical most supply promise by 0.4 days, making supply guarantees extra aggressive. Moreover, the mannequin leveraged aggregated information from a number of carriers for every vendor, enhancing its robustness and accuracy. This skill to attract insights from broader patterns considerably lowered the proportion of sellers with late charges exceeding 5%, thereby enhancing the general reliability of supply guarantees on the vendor stage.
Our enterprise objective was clear: to keep up a low late price whereas lowering the typical most supply promise to enhance competitiveness. Via our iterative improvement course of, we lowered the Avg Max by 0.9 days, considerably enhancing the competitiveness of supply home windows. These enhancements have motivated extra sellers to hitch this system. This broader adoption has additional strengthened the platform’s reliability and enriched the client expertise, making a constructive suggestions loop of belief and market competitiveness.
Constructing fashions isn’t a one-time effort; it requires steady iteration and innovation. Our success isn’t solely attributable to the fashions themselves but additionally to considerate characteristic engineering that successfully captured the complexities of vendor and provider operations. This built-in strategy has been important to attaining our outcomes.
Wanting ahead, now we have recognized a number of alternatives to additional improve the mannequin and its impression:
- Introducing the Metric for Variety of Delays: The variety of days an order is delayed has a big impression on buyer satisfaction and Internet Promoter Rating (NPS). Incorporating this metric will assist us refine our predictions and higher meet buyer expectations.
- Factoring within the Day of the Week: The day an order is positioned impacts processing and supply occasions as a result of operational realities. Together with this issue will enhance the mannequin’s accuracy and alignment with real-world situations.
- Adopting Non-Parametric Fashions: Transitioning to non-parametric fashions will improve flexibility and scalability, enabling higher dealing with of the varied dynamics in seller-carrier interactions with out the necessity to match particular distributions.
As we transfer ahead, ManoMano’s Information Scientists are dedicated to leveraging progressive and superior applied sciences to drive operational effectivity and improve buyer expertise, making certain we stay on the forefront of the trade.
