S100 Benchmark of Model Performance
-
Test Board: S100
-
Range: Models under the
samples/ucp_tutorial/dnn/ai_benchmarkpath in the OE package -
Runtime Environment: Linux
MODEL ACCURACY
| MODEL NAME | INPUT SIZE | ACCURACY | Dataset |
|---|---|---|---|
| ResNet50 | 1x3x224x224 | Top1: 0.7704(FLOAT)/0.7665(INT8) | ImageNet |
| GoogleNet | 1x3x224x224 | Top1: 0.7018(FLOAT)/0.6998(INT8) | ImageNet |
| EfficientNet_Lite1 | 1x240x240x3 | Top1: 0.7652(FLOAT)/0.7614(INT8) | ImageNet |
| EfficientNet_Lite2 | 1x260x260x3 | Top1: 0.7734(FLOAT)/0.7697(INT8) | ImageNet |
| EfficientNet_Lite3 | 1x280x280x3 | Top1: 0.7917(FLOAT)/0.7896(INT8) | ImageNet |
| EfficientNet_Lite4 | 1x300x300x3 | Top1: 0.8063(FLOAT)/0.8043(INT8) | ImageNet |
| Vargconvnet | 1x3x224x224 | Top1: 0.7793(FLOAT)/0.7770(INT8) | ImageNet |
| Efficientnasnet_m | 1x3x300x300 | Top1: 0.7935(FLOAT)/0.7923(INT8) | ImageNet |
| Efficientnasnet_s | 1x3x280x280 | Top1: 0.7441(FLOAT)/0.7524(INT8) | ImageNet |
| ResNet18 | 1x3x224x224 | Top1: 0.7170(FLOAT)/0.7157(INT8) | ImageNet |
| YOLOv2_Darknet19 | 1x3x608x608 | [IoU=0.50:0.95]= 0.2760(FLOAT)/0.2700(INT8) | COCO |
| YOLOv3_Darknet53 | 1x3x416x416 | [IoU=0.50:0.95]= 0.3370(FLOAT)/0.3360(INT8) | COCO |
| YOLOv5x_v2.0 | 1x3x672x672 | [IoU=0.50:0.95]= 0.4810(FLOAT)/0.4670(INT8) | COCO |
| Centernet_resnet101 | 1x3x512x512 | [IoU=0.50:0.95]= 0.3420(FLOAT)/0.3270(INT8) | COCO |
| YOLOv3_VargDarknet | 1x3x416x416 | [IoU=0.50:0.95]= 0.3280(FLOAT)/0.3270(INT8) | COCO |
| Deeplabv3plus_efficientnetb0 | 1x3x1024x2048 | mIoU: 0.7630(FLOAT)/0.7571(INT8) | Cityscapes |
| Fastscnn_efficientnetb0 | 1x3x1024x2048 | mIoU: 0.6997(FLOAT)/0.6914(INT8) | Cityscapes |
| Deeplabv3plus_efficientnetm1 | 1x3x1024x2048 | mIoU: 0.7794(FLOAT)/0.7754(INT8) | Cityscapes |
| Deeplabv3plus_efficientnetm2 | 1x3x1024x2048 | mIoU: 0.7882(FLOAT)/0.7854(INT8) | Cityscapes |
| Bev_gkt_mixvargenet_multitask | image: 6x3x512x960 points(0-8): 6x64x64x2 | NDS: 0.2810(FLOAT)/0.2781(INT8) MeanIOU: 0.4852(FLOAT)/0.4839(INT8) mAP: 0.1990(FLOAT)/0.1990(INT8) | Nuscenes |
| Bev_ipm_4d_efficientnetb0_multitask | image: 6x3x512x960 points: 6x128x128x2 prev_feat: 1x164x28x128 prev_point: 1x128x128x2 | NDS: 0.3722(FLOAT)/0.3730(INT8) MeanIOU: 0.5287(FLOAT)/0.5388(INT8) mAP: 0.2201(FLOAT)/0.2215(INT8) | Nuscenes |
| Bev_ipm_efficientnetb0_multitask | image: 6x3x512x960 points: 6x128x128x2 | NDS: 0.3055(FLOAT)/0.3041(INT8) MeanIOU: 0.5145(FLOAT)/0.5105(INT8) mAP: 0.2170(FLOAT)/0.2163(INT8) | Nuscenes |
| Bev_lss_efficientnetb0_multitask | image: 6x3x256x704 points(0&1): 10x128x128x2 | NDS: 0.3006(FLOAT)/0.3003(INT8) MeanIOU: 0.5180(FLOAT)/0.5171(INT8) mAP: 0.2061(FLOAT)/0.2046(INT8) | Nuscenes |
| Detr3d_efficientnetb3 | coords(0-3): 6x4x256x2 image: 6x3x512x1408 masks: 1x4x256x24 | NDS: 0.3304(FLOAT)/0.3309(INT8) mAP: 0.2753(FLOAT)/0.2743(INT8) | Nuscenes |
| Petr_efficientnetb3 | image: 6x3x512x1408 pos_embed: 1x96x44x256 | NDS: 0.3765(FLOAT)/0.3745(INT8) mAP: 0.3038(FLOAT)/0.2937(INT8) | Nuscenes |
| Bevformer_tiny_resnet50_detection | img: 6x3x480x800 prev_bev: 1x2500x256 prev_bev_ref: 1x50x50x2 queries_rebatch_grid: 6x20x32x2 restore_bev_grid: 1x100x50x2 reference_points_rebatch: 6x640x4x2 bev_pillar_counts: 1x2500x1 | NDS: 0.3713(FLOAT)/0.3700(INT8) mAP: 0.2673(FLOAT)/0.2646(INT8) | Nuscenes |
| Flashocc_henet_lss_occ3d_nuscenes | img: 6x3x512x960 points: 10x128x128x2 points_depth: 10x128x128x2 | mIoU: 0.3675(FLOAT)/0.3687(INT8) | Nuscenes |
| Horizon_swin_transformer | 1x3x224x224 | Top1: 0.8024(FLOAT)/0.7988(INT8) | ImageNet |
| Vargnetv2 | 1x3x224x224 | Top1: 0.7342(FLOAT)/0.7328(INT8) | ImageNet |
| Vit_small | 1x3x224x224 | Top1: 0.7950(FLOAT)/0.7921(INT8) | ImageNet |
| Centerpoint_pointpillar | points: 300000x5 voxel_feature: 1x5x20x40000 coors: 40000x4 | NDS: 0.5832(FLOAT)/0.5816(INT8) mAP: 0.4804(FLOAT)/0.4784(INT8) | Nuscenes |
| Detr_efficientnetb3 | 1x3x800x1333 | [IoU=0.50:0.95]= 0.3720(FLOAT)/0.3604(INT8) | MS COCO |
| Detr_resnet50 | 1x3x800x1333 | [IoU=0.50:0.95]= 0.3569(FLOAT)/0.3152(INT8) | MS COCO |
| FCOS3D_efficientnetb0 | 1x3x512x896 | NDS: 0.3061(FLOAT)/0.3030(INT8) mAP: 0.2133(FLOAT)/0.2069(INT8) | nuscenes |
| Fcos_efficientnetb0 | 1x3x512x512 | [IoU=0.50:0.95]= 0.3626(FLOAT)/0.3551(INT8) | MS COCO |
| Ganet_mixvargenet | 1x3x320x800 | F1Score: 0.7949(FLOAT)/0.7883(INT8) | CuLane |
| Deformable_detr_resnet50 | 1x3x800x1333 | [IoU=0.50:0.95]= 0.4413(FLOAT)/0.4194(INT8) | MS COCO |
| Stereonetplus_mixvargenet | 2x3x544x960 | EPE: 1.1270(FLOAT)/1.1341(INT8) | SceneFlow |
| Centerpoint_mixvargnet_multitask | points: 300000x5 voxel_feature: 1x5x20x40000 coors: 40000x4 | NDS: 0.5809(FLOAT)/0.5753(INT8) MeanIOU: 0.9128(FLOAT)/0.9121(INT8) mAP: 0.4727(FLOAT)/0.4626(INT8) | Nuscenes |
| Unet_mobilenetv1 | 1x3x1024x2048 | mIoU: 0.6801(FLOAT)/0.6767(INT8) | Cityscapes |
| Densetnt_vectornet | goals_2d: 30x1x2048x2 goals_2d_mask: 30x1x2048x1 instance_mask: 30x1x96x1 lane_feat: 30x9x64x11 traj_feat: 30x19x32x9 | minFDA: 1.2975(FLOAT)/1.3021(INT8) | Argoverse 1 |
| Maptroe_henet_tinym_bevformer | img: 6x3x480x800 osm_mask: 1x1x50x100 queries_rebatch_grid: 6x20x100x2 restore_bev_grid: 1x100x100x2 reference_points_rebatch: 6x2000x4x2 bev_pillar_counts: 1x5000x1 | mAP: 0.6633(FLOAT)/0.6564(INT8) | Nuscenes |
| Qcnet_oe | valid_mask: 1x30x10 valid_mask_a2a: 1x10x30x30 agent_type: 1x30x1 x_a_cur: 1x1x30x1,1x1x30x1,1x1x30x1,1x1x30x1 r_pl2a_cur: 1x1x30x80,1x1x30x80,1x1x30x80 r_t_cur: 1x1x30x6,1x1x30x6,1x1x30x6,1x1x30x6 r_a2a_cur: 1x1x30x30,1x1x30x30,1x1x30x30 x_a_mid_emb: 1x30x2x128 x_a: 1x30x6x128 pl_type,is_intersection: 1x80 r_pl2pl: 1x1x80x80,1x1x80x80,1x1x80x80 r_pt2pl: 1x1x80x50,1x1x80x50,1x1x80x50 mask_pl2pl: 1x80x80 magnitude,pt_type,side,mask: 1x80x50 mask_a2m: 1x30x30 mask_dst: 1x30x1 type_pl2pl: 1x80x80 | hitrate: 0.8026(FLOAT)/0.7972(INT8) | Argoverse 2 |
MODEL PERFORMANCE
-
FPS = Frames per second. This data is obtained by running the
hrt_model_exectool in multi-threaded mode. For detailed usage instructions, please refer to the the hrt_model_exec tool introduction - model performance evaluation section. -
Latency = Inference latency, with the unit of ms(milliseconds). This data is obtained by running the
hrt_model_exectool in single-threaded mode. For detailed usage instructions, please refer to the the hrt_model_exec tool introduction - model performance evaluation section.
| MODEL NAME | INPUT SIZE | Latency(ms) | FPS | FPS Configuration |
|---|---|---|---|---|
| ResNet50 | 1x3x224x224 | 1.132 | 1153.55 | thread_num:4 |
| GoogleNet | 1x3x224x224 | 0.611 | 2858.94 | thread_num:4 |
| EfficientNet_Lite1 | 1x240x240x3 | 0.595 | 3225.66 | thread_num:4 |
| EfficientNet_Lite2 | 1x260x260x3 | 0.693 | 2479.25 | thread_num:4 |
| EfficientNet_Lite3 | 1x280x280x3 | 0.812 | 1899.15 | thread_num:4 |
| EfficientNet_Lite4 | 1x300x300x3 | 1.058 | 1302.93 | thread_num:4 |
| Vargconvnet | 1x3x224x224 | 0.953 | 1454.34 | thread_num:4 |
| Efficientnasnet_m | 1x3x300x300 | 0.954 | 1455.45 | thread_num:4 |
| Efficientnasnet_s | 1x3x280x280 | 0.574 | 3210.04 | thread_num:4 |
| ResNet18 | 1x3x224x224 | 0.648 | 2565.83 | thread_num:4 |
| YOLOv2_Darknet19 | 1x3x608x608 | 4.676 | 227.57 | thread_num:4 |
| YOLOv3_Darknet53 | 1x3x416x416 | 5.034 | 210.93 | thread_num:4 |
| YOLOv5x_v2.0 | 1x3x672x672 | 16.354 | 62.35 | thread_num:4 |
| Centernet_resnet101 | 1x3x512x512 | 5.654 | 186.60 | thread_num:4 |
| YOLOv3_VargDarknet | 1x3x416x416 | 3.568 | 305.23 | thread_num:4 |
| Deeplabv3plus_efficientnetb0 | 1x3x1024x2048 | 6.920 | 150.82 | thread_num:4 |
| Fastscnn_efficientnetb0 | 1x3x1024x2048 | 4.343 | 246.34 | thread_num:4 |
| Deeplabv3plus_efficientnetm1 | 1x3x1024x2048 | 11.233 | 91.48 | thread_num:4 |
| Deeplabv3plus_efficientnetm2 | 1x3x1024x2048 | 15.829 | 64.43 | thread_num:4 |
| Bev_gkt_mixvargenet_multitask | image: 6x3x512x960 points(0-8): 6x64x64x2 | 15.072 | 68.78 | thread_num:4 |
| Bev_ipm_4d_efficientnetb0_multitask | image: 6x3x512x960 points: 6x128x128x2 prev_feat: 1x164x28x128 prev_point: 1x128x128x2 | 9.468 | 111.83 | thread_num:4 |
| Bev_ipm_efficientnetb0_multitask | image: 6x3x512x960 points: 6x128x128x2 | 9.212 | 114.94 | thread_num:4 |
| Bev_lss_efficientnetb0_multitask | image: 6x3x256x704 points(0&1): 10x128x128x2 | 5.865 | 186.15 | thread_num:4 |
| Detr3d_efficientnetb3 | coords(0-3): 6x4x256x2 image: 6x3x512x1408 masks: 1x4x256x24 | 31.178 | 32.51 | thread_num:4 |
| Petr_efficientnetb3 | image: 6x3x512x1408 pos_embed: 1x96x44x256 | 48.370 | 20.86 | thread_num:4 |
| Bevformer_tiny_resnet50_detection | img: 6x3x480x800 prev_bev: 1x2500x256 prev_bev_ref: 1x50x50x2 queries_rebatch_grid: 6x20x32x2 restore_bev_grid: 1x100x50x2 reference_points_rebatch: 6x640x4x2 bev_pillar_counts: 1x2500x1 | 32.142 | 31.54 | thread_num:4 |
| Flashocc_henet_lss_occ3d_nuscenes | img: 6x3x512x960 points: 10x128x128x2 points_depth: 10x128x128x2 | 11.132 | 92.86 | thread_num:6 |
| Horizon_swin_transformer | 1x3x224x224 | 3.391 | 321.81 | thread_num:4 |
| Vargnetv2 | 1x3x224x224 | 0.494 | 4384.88 | thread_num:4 |
| Vit_small | 1x3x224x224 | 1.955 | 596.83 | thread_num:4 |
| Centerpoint_pointpillar | points: 300000x5 voxel_feature: 1x5x20x40000 coors: 40000x4 | 13.048 | 121.67 | thread_num:4 |
| Detr_efficientnetb3 | 1x3x800x1333 | 18.902 | 53.79 | thread_num:4 |
| Detr_resnet50 | 1x3x800x1333 | 24.544 | 41.29 | thread_num:4 |
| FCOS3D_efficientnetb0 | 1x3x512x896 | 2.691 | 450.63 | thread_num:4 |
| Fcos_efficientnetb0 | 1x3x512x512 | 1.326 | 1105.46 | thread_num:4 |
| Ganet_mixvargenet | 1x3x320x800 | 0.919 | 1578.04 | thread_num:4 |
| Deformable_detr_resnet50 | 1x3x800x1333 | 200.407 | 5.00 | thread_num:6 |
| Stereonetplus_mixvargenet | 2x3x544x960 | 4.766 | 223.98 | thread_num:4 |
| Centerpoint_mixvargnet_multitask | points: 300000x5 voxel_feature: 1x5x20x40000 coors: 40000x4 | 10.370 | 185.18 | thread_num:4 |
| Unet_mobilenetv1 | 1x3x1024x2048 | 1.611 | 812.13 | thread_num:4 |
| Densetnt_vectornet | goals_2d: 30x1x2048x2 goals_2d_mask: 30x1x2048x1 instance_mask: 30x1x96x1 lane_feat: 30x9x64x11 traj_feat: 30x19x32x9 | 6.927 | 156.09 | thread_num:4 |
| Maptroe_henet_tinym_bevformer | img: 6x3x480x800 osm_mask: 1x1x50x100 queries_rebatch_grid: 6x20x100x2 restore_bev_grid: 1x100x100x2 reference_points_rebatch: 6x2000x4x2 bev_pillar_counts: 1x5000x1 | 13.856 | 74.28 | thread_num:4 |
| Qcnet_oe | valid_mask: 1x30x10 valid_mask_a2a: 1x10x30x30 agent_type: 1x30x1 x_a_cur: 1x1x30x1,1x1x30x1,1x1x30x1,1x1x30x1 r_pl2a_cur: 1x1x30x80,1x1x30x80,1x1x30x80 r_t_cur: 1x1x30x6,1x1x30x6,1x1x30x6,1x1x30x6 r_a2a_cur: 1x1x30x30,1x1x30x30,1x1x30x30 x_a_mid_emb: 1x30x2x128 x_a: 1x30x6x128 pl_type,is_intersection: 1x80 r_pl2pl: 1x1x80x80,1x1x80x80,1x1x80x80 r_pt2pl: 1x1x80x50,1x1x80x50,1x1x80x50 mask_pl2pl: 1x80x80 magnitude,pt_type,side,mask: 1x80x50 mask_a2m: 1x30x30 mask_dst: 1x30x1 type_pl2pl: 1x80x80 | 5.094 | 220.29 | thread_num:4 |