1、Real-world Insights from Mining Retail E-Commerce Data,Ronny Kohavi, Ph.D. Vice President, Business IntelligenceBlue Martini SoftwareSan Mateo, CA, ,May 22, 2003,Goals,Give you a feel for what e-commerce data looks like Show interesting insights with fun teasersfrom Blue Martini customers data Show

2、things that worked well for us, including architecture and powerful visualizations Next week: share more detailed data mining lessons and challenges (Rajesh Parekh),Agenda,Overview of architecture Usability Web site traffic Timeout Searches, referrers Micro-conversions and utilizing real-estate E-ma

3、il campaigns Multi-channel analysis Cross-sells / Associations Classification Summary,The Vision in 1998,In July of 1998, I gave an invited talk at ICML titled Crossing the Chasm: From Academic Machine Learning to Commercial Data Mining/users/ronnyk/chasm.pdf Most talks ha

4、ve one key slide (some have zero ) The key slide was the following slide, which guided the design of the data mining architecture at Blue Martini software,Key Slide in Crossing the Chasm,Our CEO did this once before,Vertical:e-commerce retail,-,Business Data Definition (Enterprise Desktop, Remote De

5、sktop),Customer Interactions (Web, campaigns, Call Center, Wireless, POS),Analysis (Reporting, Analytics, Visualizations, OLAP),Integrated Architecture,Stage Data,Deploy Results,Build Data Warehouse (DSSGen),marketplace,Advantages of Architecture,It is well documented that “80% of the time spent in

6、knowledge discovery is spent on data preparation” Our architecture shares enough meta data and there is enough domain knowledge to cut that dramatically Clickstreams Store from the application server layer to the DB (no need to load from flat files on multiple web servers, conflate, and sessionize)

7、Collect additional information (screen resolution, local time) Tie all activities (registrations, orders) to sessions Log high level “Business Events,” including cart activities, search information, form errors More information in Integrating E-Commerce and Data Mining: Architecture and Challenges,

8、ICDM 01 Available at /users/ronnyk,Usability Form Errors,This was the Bluefly home page Looking at form errors logged by our architecture, we saw thousands of errors every day on this page Any guesses?,Improved Home Page,This is the new Bluefly home page Search box added E

9、-mail box clearly marked as email As with many insights, hindsight is 20/20 The hard part is collecting the right information and reporting on it,Bot Detection,Bots are automated programs, sometimes called crawlers/robots Examples: search engines, shopping bots, performance monitors Significant traf

10、fic may be generated by bots Can you guess what percentage of sessions are generated by bots?,23% at MEC (outdoor gear),40% at Debenhams,Without bot removal, your metrics willbe inaccurate We find about 150 different bot families on most sites. Very challenging problem!,Example: Web Traffic,Weekends

11、,Sept-11 Note significant drop in human traffic, not bot traffic,Registration at Search Engine sites,Internal Perfor-mance bot,Heat Maps for Day-of-Week (Same Data),Use color to show an additional dimension Green is low traffic Yellow is medium traffic Red is high traffic The power of visualizations

12、 Weekends are very slow Friday is slow Patterns Sept 11 in green Reduced traffic after Sept 11(yellow above Sept 11) Sept 3 Labor day in green,Browsing hours,Traffic by hour (server time)Lines show two consecutive weeks What do you think it looks like? How stable is it across domains/geographies?,CS

13、T,GMT,EST,Tokyo,Drill-Down to Hour,Same heat map idea applies to hourly patterns In this case hourly traffic to a web site Note Sept 11 effect and its effect for rest of week,Site down at critical hour,Teaser,Here is a similar heatmap Interestingly, the white square (no traffic) appeared on many sit

14、es But not in Phoenix, AZ servers Why?,Site down?,Teaser,We found that people purchase hours after visiting the site,Session Timeout,Catledge and Pitkow in a well referenced paper determined that the “optimal” session timeout for analysis should be 25.5 minutes How many visitors at Debenhams Added p

15、roduct to shopping cart Waited over 25.5 minutes Came back to the site inthe next 3 hours?,Searches,Architecture records every search and the number of results Top searched keywords (percent of searches) Empty search string (3.9%)returns over 160 results GPS (1.2%) sunglasses (0.8%) Top failed keywo

16、rds in the product category (percent of failed searches) gift certificate(s) (0.98%)(already implemented since study) arcteryx (0.44%) bear spray (0.44%) pedometer (0.37%) stroller(s) (0.36%),Synonyms,At Publix, an online grocer in the southeast, Bath Tissue was among the top selling assortments Top

17、 failed search?,Toilet Paper,Search Effectiveness at MEC,Customers that search are worth two times as much as customers that do not search Failed searches hurt sales,Visit,Search(64% successful),No Search,Last Search Succeeded,Last Search Failed,10%,90%,70%,30%,Referrers at Debenhams,Top Referrers M

18、SN (including search and shopping) Average purchase per visit = X Google Average purchase per visit = 1.8X AOL search Average purchase per visit = 4.8X,Understand abandonment and conversions Not just visitor to purchaser, but also the micro-conversions Shopping Cart Abandonment 62% =55% + 45% * 17%

19、Excellent opportunityto identify problematicsteps in processes andimprove Also a good way to identify abandoned products, send targeted e-mails if those products are on sale,Micro-Conversion Rates at Debenhams,Page Effectiveness Percentage of visits clicking on different links,Top Links followed fro

20、m the Welcome Page:Revenue per session associated with visits,Note how effective physical catalog item #s are,Teaser - High Conversion Rates,Product Conversion Rate is the ratio of product purchases to product views High can conversion rates be over 100%,Conversion rates are high because Call Center

21、 (orders but no views) Automatic reordering (send me the medicine every month) Bundles (you view X, you get Y for free) Wizard (at Virgin Wines, they mix you a case; most people dont even look at the details) Quantities over 1 (question of exact definition of conversion),Teaser - Privacy,92% of Amer

22、icans are concerned (67% very concerned) about the misuse of their personal information on the Internet. - FTC Report, May 2000 86% of executives dont know how many customers view their privacy policies. - Forrester Report, November 2000 Q: What percentage of visitors read the privacy statement?,A:

23、Less than 0.3%,Direct Mail Campaigns (Why Spam),Assumptions: Response rate: 3%(This is 0.6% for credit-card solicitations now, but were going to send a wonderful offer for our Widget and get 3% response) Average revenue per response: $100 Profit margin: 20%(after all costs, including handling return

24、s, shipping, etc.) To breakeven, how much should the offer cost per person? Think about: creative design costs, letter, brochure, outer envelope, reply envelope, stamp, per-person cost when purchasing list,Cost should be less than 60 cents! 3%*$100*20% = $0.60 Obviously, its not an easy businessThat

25、s why e-mail spam are so “cost effective”,Campaign Analysis - Debenhams,Analyze the effectiveness of campaigns,Multi Channel Analysis,If we define a multi channel customer to have shopped on the web and at a store How much more do multi channel customers spend at over single channel customers?,Multi

26、channel customers spend 72% more per year than single channel customers - State of Retailing Online, ,More than twice as muchfor customers with two or more purchases(you cant be multi-channel if you havent shopped twice).,Channels by Num Purchases,The following graph shows that for each know

27、n number of purchases, the web-channel-only customer is better,Therefore, our intuition tells us that the web channel is the best channel, right?,Wrong!,Bug?,Multi-channel customers have higher total spending This is an example of Simpsons paradox,Simpsons Paradox,A woman sues Stanford for sex bias

28、She shows that the school admits 70% of males but only 56% of females Stanford agrees with these percentages Shows that in every department they accept a higher percentage of females than males What is amazing is that this can happen What is more amazing is that it happened in practice,Subtle Differ

29、ence in Conversation,Alice to Bob: Im applying to Stanford next year Bob to Alice: Sorry to hear that; I know theyre accepting more males than females,Alice to Bob: Im applying for department X at Stanford next year Bob to Alice: Lucky you, I know theyre accepting more females than males in departme

30、nt X,VS,And it doesnt matter what X is!,Here is a Simplified Version,100 customers,200 customers,30 customers,300 customers,Blue web channelGreen multi channel with web,The web channel dominates the multi-channel with webin both 2-purchases and 5 purchases,2,5,Total spending,Purchases,Product Affini

31、ties,Which products sell well together Discovered using the association algorithm For closing the loop, associations can be used to make cross-sell recommendations at the website,Product Affinities at MEC,Minimum support for the associations is 80 customers Confidence: 37% of people who purchased Or

32、bit Sleeping Pad also purchased Orbit Stuff Sack Lift: People who purchased Orbit Sleeping Pad were 222 times more likely to purchase the Orbit Stuff Sack compared to the general population,Product,Association,Lift Confidence,Orbit Sleeping Pad,Cygnet Sleeping Bag,Aladdin 2 Backpack,Primus Stove,Orb

33、it Stuff Sack,Website Recommended Products,222 37%,Bambini Tights Childrens,Bambini Crewneck Sweater Childrens,195 52%,Yeti Crew Neck Pullover Childrens,Beneficial Ts Organic Long Sleeve T-Shirt Kids,Silk Crew Womens,Silk Long Johns Womens,304 73%,Micro Check Vee Sweater,Volant Pants,Composite Jacke

34、t,Cascade Entrant Overmitts,Polartec 300 Double Mitts,51 48%,Volant Pants,Windstopper Alpine Hat,Tremblant 575 Vest Womens,Product Affinities at Debenhams,Minimum support for the associations is 50 customers Confidence: 41% of people who purchased Fully Reversible Mats also purchased Egyptian Cotton

35、 Towels Lift: People who purchased Fully Reversible Mats were 456 times more likely to purchase the Egyptian Cotton Towels compared to the general population,Building The Customer Signature,Building a customer signature is a significant effort, but well worth the effort A signature summarizes custom

36、er or visitor behavior across hundreds of attributes, many which are specific to the site Once a signature is built, it can be used to answer many questions. The mining algorithms will pick the most important attributes for each question Example attributes computed: Total Visits and Sales Revenue by

37、 Product Family Revenue by Month Customer State and Country Recency, Frequency, Monetary Latitude/Longitude from the Customers Postal Code,Migration Study - MEC,Oct 2001 Mar 2002,Apr 2002 Sep 2002,Migrators,Spent $1 to $200,Spent over $200,Spent over $200,Spent under $200,(5.5%),(94.5%),Customers wh

38、o migrated from low spenders in one 6 month period to high spenders in the following 6 month period,Key Characteristics of Migrators at MEC,During October 2001 March 2002 (Initial 6 months) Purchased at least $70 of merchandise Purchased at least twice Largest single order was at least $40 Used free

39、 shipping, not express shipping Live over 60 aerial kilometers from an MEC retail store Bought from these product families, such as socks, t-shirts, and accessories Customers who purchased shoulder bags and child carriers were LESS LIKELY to migrate,Recommendation: Score light spending customers bas

40、ed on their likelihood of migrating and market to high scorers.,Customer Locations Relative to Retail Stores,Map of Canada with store locations.,Heavy purchasing areas away from retail stores can suggest new retail store locations,No stores in several hot areas: MEC is building a store in Montreal r

41、ight now.,Distance From Nearest Store (MEC),People farther away from retail stores spend more on average Account for most of the revenues,Other Results at MEC (See Appendix),Free shipping changed to flat-fee (C$6 flat charge) Orders - down 9.5% Total sales - up 6.5% Gear Swap (buy/sell used gear) Vi

42、sit-to-Purchase very low: 0.34% vs. 2.1% for non gear-swap However, these visitors converted to purchasing customers (over multiple visits) at a rate 62% higher than visitors who never visited gear swap! Visits where an FYI page (For-Your-Information) page was viewed had a Visit-to-Purchase conversi

43、on of 7.1%,Other Results at Debenhams (See Appendix),People who got the timeout page for a high percentage of their sessions are less likely to migrate (to heavy spenders) Revenue due to wedding list item purchases is clearly affected by summer weather Weddings are more common in the summer in the U

44、K In June/July, 65% of revenues were generated through the wedding list,Summary (I),E-commerce matches the needs of data mining Huge datasets (both rows and columns) Clean data (collected electronically) Very actionable (easy to do controlled experiments) Easy to measure return-on-investment Having

45、a unified architecture (collection, transformation, analysis) saves much of the transformations needed (the 80% factor) and provides access to more data Customers need to crawl before they walk before they run. Must have simple reports,Summary (2),Focused on specific vertical e-commerce retail Enabl

46、ed us to write out-of-the box reports Easy for clients to get initial metrics and insights Encapsulate our expertise in this domain Focuses sales force, easier to demo with right vocabulary Provide visualization to show patterns(not discussed, but useful: interactive visualization) Many lessons, bot

47、h at the business level and at the more data mining technical level to be reviewed by Rajesh Parekh,Resources,WEBKDD workshops Mining E-commerce Data, the Good, the Bad, and the Ugly, invited talk at KDD 2001 industrial track Mining Customer Data, Etail CRM Summit, 2002 Integrating E-Commerce and Da

48、ta Mining: Architecture and Challenges, ICDM 2001 E-metrics Study providing stats for multiple sites, Dec 2001 Applications of Data Mining to Electronic Commerce, special issue of Data Mining and Knowledge Discovery journal Real World Performance of Association Rule Algorithms, KDD2001 - case studie

49、s, live demo,Appendix,Here are additional slides with some interesting insights,RFM Analysis,RFM Recency, Frequency, MonetaryExample Insights from Debenhams Anonymous purchasers have lower average order amount Customers who have opted out of e-mail tend to have higher average order amount People in

50、the age range 30-40 and 40-50 spend more on average,RFM Analysis (Debenhams),Recency, Frequency, and Monetary calculations are used extensively in retail for customer segmentation Implemented the Arthur-Hughes RFM Cube R, F, and M scores are binned into 5 equal sized bins Each dimension is labeled 1

51、 (best) 5 (worst) Interactive visualization using Filter Charts Look at charts instead of cell-tables,Complete RFM,Recommendation Targeted marketing campaigns to convert people to repeat purchasers, assuming they did not opt-out of e-mails,Low Medium High,Low Medium High,Interacting with the RFM vis

52、ualization,Explore sub-segments with filter charts People in the age range 30-40 and 40-50 spend more on average Anonymous purchasers have lower average order amount,Average Order Amount mapped to color,Low Medium High,RFM for Debenhams Card Owners,Recommendation Send targeted email campaign since t

53、hese are Debenhams customers. Try to “awaken” them!,Low Medium High,Low Medium High,Customers who have Opted Out,Recommendation Send targeted emails to prevent email fatigue,Customers who have opted out tend to have higher average order amount,Recommendation Log changes to opt out settings and track

54、 unsubscribes to identify email fatigue,Low Medium High,Free Shipping Offer (MEC),Free shipping stopped on Aug 14, 2002 A flat $6 Canadian Dollars shipping charge introduced Express shipping at higher charge continues Observations Total sales - up 6.5% Revenue (excluding shipping and tax) - up 2.8%

55、Orders - down 9.5% Average Sales per Order up 18%,Free Shipping Offer (Cont.),The distribution shows fewerorders from low spenders(probably a good thing) No impact on rest of buyers,Fewer low spenders (= $50),Free Shipping Offer (Cont.),Breakdown of orders by shipping method More people used express

56、 shipping, probably because the delta to ship express wasnt as large (from C$6 instead of from C$0),Free/Standard Shipping,Express Shipping,Gear Swap Pages (Cont.),Gear Swap Pages (Cont.),Done,Definitions for Gear Swap Analysis,A visitor is defined as someone who is registered (MEC member) or is ide

57、ntified by a cookie Note that in the Blue Martini system a registered user will have all of his/her cookies combined into a single visitor ID Comparing visitors who viewed gear swap with those who did not Several non-bot sessions have 1 request that just visited the MEC homepage (Main/home.jsp) To g

58、et to the Gear Swap section you have to click at least twice To make a fair comparison we have excluded all 1 request sessions that just visited the MEC homepage (Main/home.jsp) from the following analysis,Distribution of Gear Swap Visitors,Visitors who viewed Gear Swap pages had a 62% higher visito

59、r to purchaser conversion ratio as compared to those who did not view Gear Swap,Visitors: X MEC members: Y Purchasing Customers: Z,Visitors: 14.3% of X MEC members: 20.8% of Y Purchasing Customers: 21.1% of Z,Visitors: 85.7% of X MEC members: 79.2% of Y Purchasing Customers: 78.9% of Z,Visitors who ever viewed Gear Swap,Visitors who never viewed