1、DEEP LEARNING IN THE FIELD OF AUTONOMOUSDRIVINGAN OUTLINE OF THE DEPLOYMENT PROCESS FOR ADAS AND ADAlexander Frickenstein, 3/17/2019AUTONOMOUS DRIVING AT BMWGTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW| 03/20/19Page 2AUTONOMOUS DRIVING AT BMWBMW Autonomous Driving Campus in

2、 Unterschleiheim (Munich), established in 20171400 Employees incl. Partners (Sensor-processing, Data-Analytics, ML, Driving-Strategy,HW-Architecture) 81 Feature teams (incl. Partners), working in 2 weekly sprints (LESS)30 PhDsRaw data are good dataBMW AD research fleet consist of 85 cars collecting

3、2TB/h per car High resolution sensor data, like LIDAR, Camera-Unknown Author- Insight into three PhD-projects, which are drivenby the AD strategy at BMWGTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 3CONTENT Introduction: Design Process of ML-Applications for A

4、D Exemplary projects; which are driven by the AD strategy at BMW1.Fine-Grained Vehicle Representations for AD2.Self-Supervised Learning of the Drivable Area of AD3.CNN Optimization Techniques for ADGTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 4DESIGN PROCESS

5、OF ML-APPLICATIONS FOR ADGTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 5DESIGN PROCESS OF ML-APPLICATIONS FOR ADAn rough outline of the deployment*process for ADAS and ADInspired by Gajski-Kuhn chart (or Y diagram) 1Design of real-world applications include:-M

6、ultiple domains (structural, modelling, optimization)Abstraction levelsKnowledge sharing is essential for the drive of inovation (e.g. Car manufactures, technology companies)*Presented projects gives an academic insight of PhD-candidates*Datasets shown here are not used for commercial purposeFig. 1:

7、 Design Process of AD-Applications.GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 601FINE-GRAINED VEHICLE REPRESENTATIONS FOR ADBY THOMAS BAROWSKI, MAGDALENA SZCZOT AND SEBASTIAN HOUBENGTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/

8、20/19Page 7:FINE-GRAINED VEHICLE REPRESENTATIONS FOR ADBY THOMAS BAROWSKI, MAGDALENA SZCZOT AND SEBASTIAN HOUBENMotivation: a detailed understanding of complex traffic scenes -State and possible intentions of other traffic participants-Precise estimation of a vehicle pose and category-Be aware of dy

9、namic parts, e.g. Doors, Trunks-React fast and appropriate to safety critical situationsFig. 2: Exemplary 2D visualization of fragmentationlevels in the Cityscapes3 segmentation benchmark.GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 8Thomas Barowski, Magdalen

10、a Szczot and Sebastian Houben: Fine-Grained Vehicle Representations for Autonomous Driving, ITSC, 2018, 10.1109/ITSC.2018.8569930.ctionelingFINE-GRAINED VEHICLE REPRESENTATIONS FOR ADGoal:-Learn new vehicle representations by semantic segmentation Three vehicle fragmentation levels (Course Fine Full

11、):-Dividing vehicle into part areas, based on materials and fun-Embedding pose information-Annotating representations on CAD-Models-Empirically examined on VKITTY4, Cityscapes3Core idea is to extend an existing image-dataset by manuallabData generation pipeline is an adaption of thesemi-automated me

12、thod from Chabot et al. 5Fig. 2: Exemplary 2D visualization of fragmentationlevels in the Cityscapes3 segmentation benchmark.Instead, annotation is done on a set of models (3D Car Models)GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 9SEMI-AUTOMATED LABELING PI

13、PELINE (1)Vehicle Fragmentation Levels from ShapeNet (3D ModelRepository):-Different car models including WorldNet synsets (4000)-Three fragmentation levels (Coarse (4) Fine (9) Full (27)-Including classes for: Body, windows, lights, wheels, doors,roof, side, trunk, wheels, windshiels-In finer grain

14、ed representations: model needs to solvechallenging task of separation between parts that share visualcues but vary in position, e.g. individual doorsFig. 3: Visualization of the annotated CAD models 5.-Identify parts with small local visual context: representationbecomes suitable for pose estimatio

15、n with high occlusion ortruncationGTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 10SEMI-AUTOMATED LABELING PIPELINE (2)1.2.Apply well overlapping 3D bounding boxes to raw imagesSuited model is selected based on the vehicle type or dimensions of the model (L1-di

16、stance)Mesh of the 3D-model is resize to fit the bounding box and aligned to 3D spaceMesh is projected on the image plane: Resulting in a segmentation map containingfragmentation level information of the vehicle3.4.Only pixels labeled as vehicle in the respective dataset arepropagated to the image T

17、o overcome projection errors5. Results in fine-grained dense representationsFig. 4: Semi-automated labeling pipeline.GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 11FCN MODEL EXPLORATIONReimplemented FCN8 6 and VGG16 7 as backboneEnd to end training, using cro

18、ss entropy lossTrained on 4-27 classes (based on fragmentation level) Plus classes of datasetsMulti-GPU training (Kubernets and Horovod on DGX1) Full fragmentation level High resolution input imagesAim: not loosing significant accuracy in non vehicle-related background classesFig. 5: FCN8 6 with VGG

19、167 backbone.GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 12FCN MODEL EXPLORATIONExperimentIoUclassIoUnon-partsIoUpartsVKITTY ShapeNet 15Coarse61.0566.4963.64Full36.6758.2227.44Fine48.6350.8844.20Tab. 1: Segmentation results for the three fragmentation levels

20、, performed on VKITTY and Cityscapes using FCN8.GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 13Full33.5050.7821.98Cityscapes ShapeNet 15Coarse49.5648.8152.96Fine56.9366.3144.73VKITTY Baseline68.7768.77-FCN MODEL EXPLORATION VKITTY AND CITYSCAPESCoarse:Fine:Fu

21、ll:Fig. 6a: Qualitative results on VKITTY dataset for the three fragmentation levels. Fig. 6b: Qualitative results on Cityscapes dataset for the three fragmentation levels.GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 1402SELF-SUPERVISED LEARNING OF THE DRIVAB

22、LE AREA OF ADBY JAKOB MAYR, CHRISTIAN UNGER, FEDERICO TOMBARIGTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 15SELF-SUPERVISED LEARNING OF THE DRIVABLE AREA OF ADMotivation:- Automated approach for generating training data for the task of drivablearea segmentati

23、on Training Data Generator (TDG)- Acquisition of large scale datasets with associated ground-truth still poses an expensive and labor-intense problemDeterministic stereo-based approach for ground-plane detection:Fig. 7a: Automated generated data of TDG.Fig. 7b: Segmentation of DNN trained on TDG.GTC

24、 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 16Jakob Mayr, Christian Unger, Federico Tombari: Self-Supervised Learning of the Drivable Area for Autonomous Driving, iROS, 2018.WHY GROUND-PLANE DETECTION?Important aspect is the planning of safe and comfortable dri

25、ving maneuversKnowledge about the environment of the vehicle especially drivable areas (important role in ADAS and AD)e.g. road ahead/ drivable area is blocked by obstaclesParallel processing of GPUs allow frame based semantic segmentation Why Automated Data-Labeling?-Pace and cost pressureLabeling

26、is expensiveExisting datasets do not suit the desired application:o Technical aspects: e.g. field of view, mounting position, camera geometryo Environmental conditions: e.g. weather condition, time, street typesGTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 17T

27、echnical Aspect of Cityscapes: images show part of the hood, initialization of the ground-plane model including non-ground plane disparity is necessary!AUTOMATED LABELING PIPELINEBased on so-called v-disparity map 8:-Different use casesNo fine tuning of existing models requiredFig. 8: Automated labe

28、ling pipeline.GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 18CNN-MODEL EXPLORATION (1)Automatically generated data are used to train Unet and SegNet low resolution inputs (512*256 and 480*360)Models are trained only on automatically generated datasetsEvaluati

29、on is performed by using human labeled ground-truth data, e.g. Cityscape 3, Kitty 2 Drivable (road, parking) and non-drivable area (side walks, pedestrians)Observations:-Low detection in lateral directionNoisy data of TDG generate robust CNN modelDynamic objects are detected reliablyFig. 9a: SegNet

30、segmentation.Fig. 9b: U-Net segmentation.GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 19CNN-MODEL EXPLORATION (1)Robustness of model - Flipping images upsidedown:-SegNet 9UNet 10Not capable of detecting ground-planeDetects ground-planeFig. 9c: Flipped SegNet

31、segmentation.Fig. 9d: Flipped U-Net segmentation.GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 20CNN-MODEL EXPLORATION (2)CityscapesKITTYTraining Data Generator (TDG)70.8491.4985.3566.4681.8758.0786.5851.45SegNet trained on auto. TDG labels85.7576.1085.2967.56

32、90.9670.0191.3565.45Tab. 2: Segmentation results for the TDG, performed on Cityscapes 3 and Kitty 2 using U-Net 10 and SegNet 9.Performance:- Data generation:- Hand labeling Cityscape 3 19d- Using automated labeling 3.5h (CPU) not parallelized on GPU yetU-Net 10 : 10 fps on Titan X- ResNet 9.:4.4fps

33、 on Titan X Optimization of DNNs comes into account Out of the box CNNs come along with substantialdrawbacksGTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 21Unet trained on auto. TDG labels85.2992.8391.2780.0187.2581.3594.3172.70ApproachRecPrecAccIoURecPrecAccI

34、oU03CNN OPTIMIZATION TECHNIQUES FOR ADBY ALEXANDER FRICKENSTEIN, MANOJ VEMPARALA, WALTER STECHELEGTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 22CNN OPTIMIZATION TECHNIQUES FOR ADRunning example: Quantization of CNNs:-Normally, floating-point PEs is 10 less en

35、ergyefficient compared to fixed point math.The step-size between two numbers could be dynamic using floating-point numbers. This is useful feature for different kinds of layers in CNN.Closing gap between CNN compute demand and HW-accelerator is important Trend to specialized HW-accelerator, e.g. Tes

36、la-T4Fig. 10: How deployment of DNNs can be seen differently.GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 23CNN OPTIMIZATION TECHNIQUES FOR ADFig. 11: Optimization design process.GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/1

37、9Page 24RESOURCE AWARE MULTICRITERIAL OPTIMIZATIONMotivation:-Out of the box DNNs require high performance HW-accelerator:- YOLO 11 or SSD300 12 require an Nvidia Titan X to run in real-time Showing the high compute demand of those models- No, SqueezeNet 13 is really not out of the box! An 18,000 GP

38、U super-computer is used for the model explorationDeploy DNNs on memory, performance and power-consumptionconstraint embedded hardware is commonly time consuming-Fig. 11: Filter-wise pruning of convolutional layer.GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page

39、25WHY RESOURCE-AWARE MULTICRITERIAL OPTIMIZATION?Optimization techniques are going hand in hand(Filter-wise pruning and Quantization) CNN optimization depends on system,algorithmic and system level in the design processDNNS need to be highly compressed to fit the HW for AD Automotive rated memory is

40、 expensiveGood data locality is essential for low-power applications Extreme temperatures in cars (Siberia Death Valley) Active cooling obligatory?Fast deployment time is a crucial aspect foragile SW deploymentFig. 12: Pruning and quantization for efficient embedded Applications of D Proposing a Pru

41、ning and Quantization scheme forCNNsGTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 26METHOD -PRUNINGFast magnitude based 14 pruning scheme removing unimportant filterBased on a half-interval search (log2(n) Explore optimal layer-wise pruning rateVarying pruning

42、 order to generate an optimized model either with respect to the memory demand or execution timeProcess of removing weight-filter of a layer:.Identify Cost (L1-distance) of all weight-filter of a layerBased on the half-interval search remove filter, which cost is below a threshold Threshold i

43、s identified by half-interval searchRetrain model (SGD with momentum) with small learning rate Momentum should be available before pruningAs accuracy of the CNN is maintained increase the pruning rate (half-interval search)Fig. 13: Binary-search applied to prune CNN.5.6.Pruning is always applied to

44、the layer which fits the desired optimization goal bestGTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 27METHOD - QUANTIZATIONQuantization leads to a hardware friendly implementationReducing the footprint of HW-components Lowering the memory bandwidth Improving

45、the performance Floating-point PE is 10x less efficientcompared to fixed-point unitWeight and activations are brought into the fixed-pointformat with the notation -S: Sign bitIB: Integer bitFB: Fractional bitStochastic rounding is used for approximationFig. 14: Pruning and quantization applied to CN

46、N.GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 28MEMORY AND PERFORMANCE BENCHMARKFig. 15: Pruning rate and runtime of Deep Compression 14 and our approach.Fig. 16: Runtime of VGG16 7 on different HW-Accelerator.GTC 2019 - Silicon Valley| Deep Learning for Aut

47、onomous Driving at BMW | 03/20/19Page 29MEMORY AND PERFORMANCE BENCHMARKTab. 3: Performance and memory benchmark of our method applied toVGG16.GTC 2019 - Silicon Valley| Deep Learning for Autonomous Driving at BMW | 03/20/19Page 30REFERENCES12345678Grout Ian: Digital Systems Design with FPGAs and CP

48、LDs, 2008.Andreas Geiger, Philip Lenz, Raquel Urtasun, et al.: Are we ready for autonomous driving? The KITTI vision benchmark suite, CVPR, 2012. Marius Cordts, Mohamed Omran, Sebastian Ramos, et al.: The Cityscapes Dataset for Semantic Urban Scene Understanding, CVPR, 2016. Adrien Gaidon, Qiao Wang, Yohann Cabon, et al.: Virtu