1、互聯網用戶行為日志數據分析大數據課程作業(yè)吳寧時間: 2017.05.14 15:00原始數據-用戶信息USERIDGENDERBIRTHDAYEDUJOBINCOMEPROVINCECITYISCITY13F320E7B5EAD1024AC95C3B208610DB男1981高中/中專/技校企業(yè)/公司一般職員15012000元內蒙古呼和浩特unknownEA3502C3594588F0E9D5142F99C66627女1989大學本科學生無收入浙江嘉興unknown0B5F2CAF4C5DE1E3A8E68F505AE6899B女1975大專專業(yè)技術人員50018000元廣東珠海城市58594

2、26ED1A0862FAD780BEA63537AB9女1981高中/中專/技校個體戶/自由職業(yè)者5011000元安徽蕪湖城市原始數據-用戶操作日志文件Last38638L_Start2012-05-07 08-27-59T153=Pexplorer.exe=I144=W10092=V6.00.2900.5512T173=Pwps.exe=I4528=W30488=V1,0,0,2877=NKingsoft Office=CKingsoft Corp. Ltd.T1389=Pexplorer.exe=I144=W30056=V6.00.2900.5512T1399=Pwps.exe=I4528

3、=W104a6=V1,0,0,2877T1417=Pexplorer.exe=I144=W705d0=V6.00.2900.5512T1419=Pwps.exe=I4528=W104a6=V1,0,0,2877T2971=Pexplorer.exe=I144=W705d0=V6.00.2900.5512T159=PQQ.exe=I4092=W10474=V1.71.2521.290T184=PQPlus.exe=I5056=W2047a=V0, 0, 0, 0=NQ+=CTencentT194=PQQ.exe=I4092=W10556=V1.71.2521.290T222=Ppopup_QQ.

4、exe=I4092=W305d2=V1.71.2521.290T242=Pchrome.exe=I5932=U 200萬行左右數據分析目標 分析用戶經常使用的應用程序的關聯關系 用戶年齡與上網頁時間的關系 用戶年齡與在線時間的關系 學歷與上網頁時間的關系 學歷與在線時間的關系 數據分析過程l 數據解析：解析日志文件,將數據讀入預處理程序中,將需要的數據分組存入文件中.l 數據集成：將日志文件(體現同)和用戶信息表(excel) 關聯在一起 .l 數據選擇：從數據文件中提取與分析任務相關的數據l 數據分析：根據課堂所學的大數據知識,分析數據,獲得分析結果(知識)l 知識表示：用圖形直觀表示分析結

5、果.數據解析-用戶使用計算機的時間V1V2V3V4V5V60143692D264FD906F10B8ECAB0F139D177712012-05-07 12-29-51106485801EDEB72AE20FE8BD93D126EC0FBAF9118812012-05-07 12-42-49194168260238BF8BE49F536ABC8F6670B766CAA7481912012-05-07 10-11-57113861852002BF8ABFA665EA584964322AEC740E2057432012-05-07 20-20-28360360339FC64528A066AB3EE

6、8E1C90CA0B56193072012-05-07 07-32-0259720439303AF956B4521A1AE096D3F10FB662D23153782012-05-07 11-44-42241024103C3D1E536560A76B66CAC523F25838530552012-05-07 182A06576A04EEF4BE27B87FE84C7D2313502012-05-07 08-05-0660911749203DF84A064F0A0A3FC0EBB8B1A833DD5366692012-05-07 07-44-381070107數據

7、解析-用戶應用程序/網站idP0143692D264FD906F10B8ECAB0F139D1explorer.exe0143692D264FD906F10B8ECAB0F139D1360Safe.exe0143692D264FD906F10B8ECAB0F139D1360chrome.exe0143692D264FD906F10B8ECAB0F139D1liveupdate.exe0143692D264FD906F10B8ECAB0F139D1QQ.exe0143692D264FD906F10B8ECAB0F139D1iexplore.exe0143692D264FD906F10B8ECAB

8、0F139D1360leakfixer.exe0143692D264FD906F10B8ECAB0F139D1popup_QQ.exe0143692D264FD906F10B8ECAB0F139D1360sd.exe01EDEB72AE20FE8BD93D126EC0FBAF91360chrome.exe01EDEB72AE20FE8BD93D126EC0FBAF91logon.scr01EDEB72AE20FE8BD93D126EC0FBAF91QQ.exe01EDEB72AE20FE8BD93D126EC0FBAF91explorer.exe01EDEB72AE20FE8BD93D126E

9、C0FBAF91popup_QQ.exe0238BF8BE49F536ABC8F6670B766CAA7explorer.exe0238BF8BE49F536ABC8F6670B766CAA7idU0143692D264FD906F10B8ECAB0F139D10143692D264FD906F10B8ECAB0F139D1NULL0143692D264FD906F10B8ECAB0F139D1hao1230143692D264FD906F10B8ECAB0F139D1gougou0143692D264FD906F10B8ECAB0F139D1baidu0143692D264FD906F10B

10、8ECAB0F139D1aipai0143692D264FD906F10B8ECAB0F139D1qzone0143692D264FD906F10B8ECAB0F139D1qq01EDEB72AE20FE8BD93D126EC0FBAF91qq01EDEB72AE20FE8BD93D126EC0FBAF9101EDEB72AE20FE8BD93D126EC0FBAF9152bus01EDEB72AE20FE8BD93D126EC0FBAF91baidu01EDEB72AE20FE8BD93D126EC0FBAF91i01EDEB72AE20FE8BD93D126EC0FBAF91doyouhi

11、ke01EDEB72AE20FE8BD93D126EC0FBAF91haishanghua用戶-應用程序用戶-網站應用程序的關聯關系items support130242 explorer.exe 0.938257993384785130243 QQ.exe 0.845644983461962130241 explorer.exe,QQ.exe 0.797133406835722130240 popup_QQ.exe,QQ.exe 0.558985667034179130244 popup_QQ.exe 0.558985667034179130239 explorer.exe,popup_QQ

12、.exe 0.531422271223815130245 iexplore.exe 0.527012127894157130236 explorer.exe,iexplore.exe 0.482910694597574130237 iexplore.exe,QQ.exe 0.458654906284454130248 360leakfixer.exe 0.453142227122382130246 360MobileMgr.exe 0.449834619625138130249 360se.exe 0.449834619625138130221 360leakfixer.exe,explore

13、r.exe 0.439911797133407130232 360MobileMgr.exe,explorer.exe 0.437706725468578130214 360se.exe,explorer.exe 0.427783902976847130250 360tray.exe 0.426681367144432130206 360tray.exe,explorer.exe 0.416758544652701130247 rundll32.exe 0.412348401323043130197 360sd.exe,explorer.exe 0.407938257993385130251

14、360sd.exe 0.407938257993385130227 explorer.exe,rundll32.exe 0.398015435501654130215 360se.exe,QQ.exe 0.395810363836825130233 360MobileMgr.exe,QQ.exe 0.395810363836825130222 360leakfixer.exe,QQ.exe 0.390297684674752130207 360tray.exe,QQ.exe 0.379272326350606130228 QQ.exe,rundll32.exe 0.36714443219404

15、6應用程序分布圖應用相關性-余弦公示計算結果0143692D264FD906F10B8ECAB0F139D1:c(714, 1685, 77, 70, 1621)01EDEB72AE20FE8BD93D126EC0FBAF91:c(714, 1685, 1621, 77, 70)0238BF8BE49F536ABC8F6670B766CAA7:c(714, 1685, 77, 70, 109)02BF8ABFA665EA584964322AEC740E20:c(714, 1685, 1621, 70, 109)0339FC64528A066AB3EE8E1C90CA0B56:c(1685, 7

16、14, 1621, 109, 1876)03AF956B4521A1AE096D3F10FB662D23:c(1685, 714, 1876, 1621, 2185)03C3D1E536560A76B66CAC523F258385:c(714, 1685, 77, 93, 70)03D2A06576A04EEF4BE27B87FE84C7D2:c(714, 1685, 1876, 70, 1621)03DF84A064F0A0A3FC0EBB8B1A833DD5:c(714, 1685, 77, 70, 93)03E7A57A54E58BEC81CCD69D8239583B:c(714, 16

17、85, 77, 70, 2448)03ED6C135C5912CF3BD6060F43EDEDF1:c(714, 1685, 1032, 109, 1621)0496990FDBBC5B49DF47D934CFF28CFB:c(714, 2448, 77, 2446, 70)04BABE9EBBC0F48F79ABD7CB6191508E:c(714, 1685, 1621, 1032, 109)0564D287D61CDB0C9870A405E34A5E82:c(29, 2688, 587, 712, 586)06329D6F99A22DD5B296988E9F716422:c(2602,

18、2165, 1818, 2110, 2021)066906620E978664F94CD28D5E2AED02:c(1685, 714, 1621, 1876, 1032)067FC9C8D8D15B80CA775A1D9433CB48:c(714, 1685, 806, 822, 1678)06E9A101B77B2DAEBE8E57420B7E683E:c(1685, 714, 1621, 1032, 1318)074AC99B8EB791C444E70D63CD55EB10:c(714, 1685, 77, 1876, 70)0785946275B91BE0F31FAD4A225ECFC

19、2:c(714, 1685, 2495, 70, 93)078D31F088D04F0A05AD1F68791F0653:c(2448, 2446, 714, 2456, 70)07AAAC73119FFA7CF51E682D60AAD759:c(714, 1685, 371, 1245, 1419)07BBF3D62377B1FF602C823E8DA9FC4D:c(1685, 714, 1621, 1876, 2495)07C72DF2929683FCDC1EBCF48F40DDCE:c(714, 1685, 2495, 77, 93)0825DF17932DDAC75AD58D0AD5339AA0:c(1685, 714, 1621, 77, 1032)085DCFD65206B0BB8CF78072664FA04F:c(2, 3, 1685, 839, 1032)應用程序關聯度應用程序關聯性應用程序的關聯關系 用途1:應用商店可以用來做程序推薦 用戶2:應用廠家選擇在引流用戶時盡量選擇關聯度小的應用合作,避免引入同一類用戶年齡與上網頁時間的關系年齡集中在22-35