S600 Benchmark of Model Performance
-
Test Board: S600
-
Range: Models under the
samples/ucp_tutorial/dnn/ai_benchmarkpath in the OE package -
Runtime Environment: Linux
MODEL ACCURACY
| MODEL NAME | INPUT SIZE | ACCURACY | Dataset |
|---|---|---|---|
| ResNet50 | 1x3x224x224 | Top1: 0.7704(FLOAT)/0.7661(INT8) | ImageNet |
| GoogleNet | 1x3x224x224 | Top1: 0.7018(FLOAT)/0.6995(INT8) | ImageNet |
| EfficientNet_Lite1 | 1x240x240x3 | Top1: 0.7652(FLOAT)/0.7602(INT8) | ImageNet |
| EfficientNet_Lite2 | 1x260x260x3 | Top1: 0.7734(FLOAT)/0.7696(INT8) | ImageNet |
| EfficientNet_Lite3 | 1x280x280x3 | Top1: 0.7917(FLOAT)/0.7885(INT8) | ImageNet |
| EfficientNet_Lite4 | 1x300x300x3 | Top1: 0.8063(FLOAT)/0.8041(INT8) | ImageNet |
| Vargconvnet | 1x3x224x224 | Top1: 0.7793(FLOAT)/0.7765(INT8) | ImageNet |
| Efficientnasnet_m | 1x3x300x300 | Top1: 0.7935(FLOAT)/0.7927(INT8) | ImageNet |
| Efficientnasnet_s | 1x3x280x280 | Top1: 0.7441(FLOAT)/0.7512(INT8) | ImageNet |
| ResNet18 | 1x3x224x224 | Top1: 0.7170(FLOAT)/0.7159(INT8) | ImageNet |
| YOLOv2_Darknet19 | 1x3x608x608 | [IoU=0.50:0.95]= 0.2760(FLOAT)/0.2707(INT8) | COCO |
| YOLOv3_Darknet53 | 1x3x416x416 | [IoU=0.50:0.95]= 0.3370(FLOAT)/0.3370(INT8) | COCO |
| YOLOv5x_v2.0 | 1x3x672x672 | [IoU=0.50:0.95]= 0.4810(FLOAT)/0.4670(INT8) | COCO |
| Centernet_resnet101 | 1x3x512x512 | [IoU=0.50:0.95]= 0.3420(FLOAT)/0.3270(INT8) | COCO |
| YOLOv3_VargDarknet | 1x3x416x416 | [IoU=0.50:0.95]= 0.3280(FLOAT)/0.3260(INT8) | COCO |
| Deeplabv3plus_efficientnetb0 | 1x3x1024x2048 | mIoU: 0.7630(FLOAT)/0.7569(INT8) | Cityscapes |
| Fastscnn_efficientnetb0 | 1x3x1024x2048 | mIoU: 0.6997(FLOAT)/0.6909(INT8) | Cityscapes |
| Deeplabv3plus_efficientnetm1 | 1x3x1024x2048 | mIoU: 0.7794(FLOAT)/0.7756(INT8) | Cityscapes |
| Deeplabv3plus_efficientnetm2 | 1x3x1024x2048 | mIoU: 0.7882(FLOAT)/0.7854(INT8) | Cityscapes |
| Bev_gkt_mixvargenet_multitask | image: 6x3x512x960 points(0-8): 6x64x64x2 | NDS: 0.2810(FLOAT)/0.2788(INT8) MeanIOU: 0.4852(FLOAT)/0.4839(INT8) mAP: 0.1990(FLOAT)/0.1992(INT8) | Nuscenes |
| Bev_ipm_4d_efficientnetb0_multitask | image: 6x3x512x960 points: 6x128x128x2 prev_feat: 1x164x28x128 prev_point: 1x128x128x2 | NDS: 0.3722(FLOAT)/0.3723(INT8) MeanIOU: 0.5287(FLOAT)/0.5389(INT8) mAP: 0.2201(FLOAT)/0.2217(INT8) | Nuscenes |
| Bev_ipm_efficientnetb0_multitask | image: 6x3x512x960 points: 6x128x128x2 | NDS: 0.3055(FLOAT)/0.3041(INT8) MeanIOU: 0.5145(FLOAT)/0.5103(INT8) mAP: 0.2170(FLOAT)/0.2166(INT8) | Nuscenes |
| Bev_lss_efficientnetb0_multitask | image: 6x3x256x704 points(0&1): 10x128x128x2 | NDS: 0.3006(FLOAT)/0.3008(INT8) MeanIOU: 0.5180(FLOAT)/0.5172(INT8) mAP: 0.2061(FLOAT)/0.2043(INT8) | Nuscenes |
| Detr3d_efficientnetb3 | coords(0-3): 6x4x256x2 image: 6x3x512x1408 masks: 1x4x256x24 | NDS: 0.3304(FLOAT)/0.3306(INT8) mAP: 0.2753(FLOAT)/0.2742(INT8) | Nuscenes |
| Petr_efficientnetb3 | image: 6x3x512x1408 pos_embed: 1x96x44x256 | NDS: 0.3765(FLOAT)/0.3748(INT8) mAP: 0.3038(FLOAT)/0.2942(INT8) | Nuscenes |
| Bevformer_tiny_resnet50_detection | img: 6x3x480x800 prev_bev: 1x2500x256 prev_bev_ref: 1x50x50x2 queries_rebatch_grid: 6x20x32x2 restore_bev_grid: 1x100x50x2 reference_points_rebatch: 6x640x4x2 bev_pillar_counts: 1x2500x1 | NDS: 0.3713(FLOAT)/0.3700(INT8) mAP: 0.2673(FLOAT)/0.2644(INT8) | Nuscenes |
| Flashocc_henet_lss_occ3d_nuscenes | img: 6x3x512x960 points: 10x128x128x2 points_depth: 10x128x128x2 | mIoU: 0.3675(FLOAT)/0.3685(INT8) | Nuscenes |
| Horizon_swin_transformer | 1x3x224x224 | Top1: 0.8024(FLOAT)/0.7982(INT8) | ImageNet |
| Vargnetv2 | 1x3x224x224 | Top1: 0.7342(FLOAT)/0.7332(INT8) | ImageNet |
| Vit_small | 1x3x224x224 | Top1: 0.7950(FLOAT)/0.7924(INT8) | ImageNet |
| Centerpoint_pointpillar | points: 300000x5 voxel_feature: 1x5x20x40000 coors: 40000x4 | NDS: 0.5832(FLOAT)/0.5820(INT8) mAP: 0.4804(FLOAT)/0.4783(INT8) | Nuscenes |
| Detr_efficientnetb3 | 1x3x800x1333 | [IoU=0.50:0.95]= 0.3720(FLOAT)/0.3584(INT8) | MS COCO |
| Detr_resnet50 | 1x3x800x1333 | [IoU=0.50:0.95]= 0.3569(FLOAT)/0.3168(INT8) | MS COCO |
| FCOS3D_efficientnetb0 | 1x3x512x896 | NDS: 0.3061(FLOAT)/0.3022(INT8) mAP: 0.2133(FLOAT)/0.2067(INT8) | nuscenes |
| Fcos_efficientnetb0 | 1x3x512x512 | [IoU=0.50:0.95]= 0.3626(FLOAT)/0.3562(INT8) | MS COCO |
| Ganet_mixvargenet | 1x3x320x800 | F1Score: 0.7949(FLOAT)/0.7884(INT8) | CuLane |
| Deformable_detr_resnet50 | 1x3x800x1333 | [IoU=0.50:0.95]= 0.4413(FLOAT)/0.4499(INT8) | MS COCO |
| Stereonetplus_mixvargenet | 2x3x544x960 | EPE: 1.1270(FLOAT)/1.1336(INT8) | SceneFlow |
| Centerpoint_mixvargnet_multitask | points: 300000x5 voxel_feature: 1x5x20x40000 coors: 40000x4 | NDS: 0.5809(FLOAT)/0.5766(INT8) MeanIOU: 0.9128(FLOAT)/0.9126(INT8) mAP: 0.4727(FLOAT)/0.4649(INT8) | Nuscenes |
| Unet_mobilenetv1 | 1x3x1024x2048 | mIoU: 0.6801(FLOAT)/0.6764(INT8) | Cityscapes |
| Densetnt_vectornet | goals_2d: 30x1x2048x2 goals_2d_mask: 30x1x2048x1 instance_mask: 30x1x96x1 lane_feat: 30x9x64x11 traj_feat: 30x19x32x9 | minFDA: 1.2975(FLOAT)/1.2989(INT8) | Argoverse 1 |
| Maptroe_henet_tinym_bevformer | img: 6x3x480x800 osm_mask: 1x1x50x100 queries_rebatch_grid: 6x20x100x2 restore_bev_grid: 1x100x100x2 reference_points_rebatch: 6x2000x4x2 bev_pillar_counts: 1x5000x1 | mAP: 0.6633(FLOAT)/0.6569(INT8) | Nuscenes |
| Qcnet_oe | valid_mask: 1x30x10 valid_mask_a2a: 1x10x30x30 agent_type: 1x30x1 x_a_cur: 1x1x30x1,1x1x30x1,1x1x30x1,1x1x30x1 r_pl2a_cur: 1x1x30x80,1x1x30x80,1x1x30x80 r_t_cur: 1x1x30x6,1x1x30x6,1x1x30x6,1x1x30x6 r_a2a_cur: 1x1x30x30,1x1x30x30,1x1x30x30 x_a_mid_emb: 1x30x2x128 x_a: 1x30x6x128 pl_type,is_intersection: 1x80 r_pl2pl: 1x1x80x80,1x1x80x80,1x1x80x80 r_pt2pl: 1x1x80x50,1x1x80x50,1x1x80x50 mask_pl2pl: 1x80x80 magnitude,pt_type,side,mask: 1x80x50 mask_a2m: 1x30x30 mask_dst: 1x30x1 type_pl2pl: 1x80x80 | hitrate: 0.8026(FLOAT)/0.7979(INT8) | Argoverse 2 |
MODEL PERFORMANCE
-
FPS = Frames per second. This data is obtained by running the
hrt_model_exectool in multi-threaded mode. For detailed usage instructions, please refer to the the hrt_model_exec tool introduction - model performance evaluation section. -
Latency = Inference latency, with the unit of ms(milliseconds). This data is obtained by running the
hrt_model_exectool in single-threaded mode. For detailed usage instructions, please refer to the the hrt_model_exec tool introduction - model performance evaluation section.
Before performance evaluation, please refer to the following method to set the number of threads in the thread pool for CPU operator inference in model inference to 12 via environment variables:
| MODEL NAME | INPUT SIZE | Latency(ms) | FPS | FPS Configuration |
|---|---|---|---|---|
| ResNet50 | 1x3x224x224 | 0.587 | 6519.27 | thread_num:10 |
| GoogleNet | 1x3x224x224 | 0.379 | 18480.92 | thread_num:12 |
| EfficientNet_Lite1 | 1x240x240x3 | 0.382 | 17998.63 | thread_num:12 |
| EfficientNet_Lite2 | 1x260x260x3 | 0.447 | 14056.08 | thread_num:10 |
| EfficientNet_Lite3 | 1x280x280x3 | 0.512 | 11572.69 | thread_num:10 |
| EfficientNet_Lite4 | 1x300x300x3 | 0.656 | 8327.86 | thread_num:10 |
| Vargconvnet | 1x3x224x224 | 0.523 | 12423.31 | thread_num:12 |
| Efficientnasnet_m | 1x3x300x300 | 0.576 | 9260.61 | thread_num:10 |
| Efficientnasnet_s | 1x3x280x280 | 0.376 | 19348.62 | thread_num:12 |
| ResNet18 | 1x3x224x224 | 0.384 | 14316.09 | thread_num:12 |
| YOLOv2_Darknet19 | 1x3x608x608 | 2.229 | 1987.00 | thread_num:8 |
| YOLOv3_Darknet53 | 1x3x416x416 | 2.453 | 1752.26 | thread_num:8 |
| YOLOv5x_v2.0 | 1x3x672x672 | 8.202 | 494.50 | thread_num:8 |
| Centernet_resnet101 | 1x3x512x512 | 2.446 | 1786.98 | thread_num:8 |
| YOLOv3_VargDarknet | 1x3x416x416 | 1.628 | 2673.33 | thread_num:8 |
| Deeplabv3plus_efficientnetb0 | 1x3x1024x2048 | 2.952 | 1468.62 | thread_num:8 |
| Fastscnn_efficientnetb0 | 1x3x1024x2048 | 1.741 | 2647.19 | thread_num:8 |
| Deeplabv3plus_efficientnetm1 | 1x3x1024x2048 | 5.126 | 818.14 | thread_num:8 |
| Deeplabv3plus_efficientnetm2 | 1x3x1024x2048 | 7.330 | 561.92 | thread_num:8 |
| Bev_gkt_mixvargenet_multitask | image: 6x3x512x960 points(0-8): 6x64x64x2 | 6.911 | 605.09 | thread_num:8 |
| Bev_ipm_4d_efficientnetb0_multitask | image: 6x3x512x960 points: 6x128x128x2 prev_feat: 1x164x28x128 prev_point: 1x128x128x2 | 4.615 | 940.42 | thread_num:8 |
| Bev_ipm_efficientnetb0_multitask | image: 6x3x512x960 points: 6x128x128x2 | 4.499 | 967.65 | thread_num:8 |
| Bev_lss_efficientnetb0_multitask | image: 6x3x256x704 points(0&1): 10x128x128x2 | 3.075 | 1454.60 | thread_num:8 |
| Detr3d_efficientnetb3 | coords(0-3): 6x4x256x2 image: 6x3x512x1408 masks: 1x4x256x24 | 15.168 | 258.88 | thread_num:8 |
| Petr_efficientnetb3 | image: 6x3x512x1408 pos_embed: 1x96x44x256 | 21.444 | 187.70 | thread_num:8 |
| Bevformer_tiny_resnet50_detection | img: 6x3x480x800 prev_bev: 1x2500x256 prev_bev_ref: 1x50x50x2 queries_rebatch_grid: 6x20x32x2 restore_bev_grid: 1x100x50x2 reference_points_rebatch: 6x640x4x2 bev_pillar_counts: 1x2500x1 | 14.164 | 276.44 | thread_num:8 |
| Flashocc_henet_lss_occ3d_nuscenes | img: 6x3x512x960 points: 10x128x128x2 points_depth: 10x128x128x2 | 5.519 | 742.29 | thread_num:8 |
| Horizon_swin_transformer | 1x3x224x224 | 1.694 | 2556.99 | thread_num:8 |
| Vargnetv2 | 1x3x224x224 | 0.332 | 19838.28 | thread_num:12 |
| Vit_small | 1x3x224x224 | 1.092 | 4245.19 | thread_num:8 |
| Centerpoint_pointpillar | points: 300000x5 voxel_feature: 1x5x20x40000 coors: 40000x4 | 7.530 | 1067.16 | thread_num:16 |
| Detr_efficientnetb3 | 1x3x800x1333 | 8.499 | 390.55 | thread_num:8 |
| Detr_resnet50 | 1x3x800x1333 | 10.795 | 317.51 | thread_num:8 |
| FCOS3D_efficientnetb0 | 1x3x512x896 | 1.475 | 3567.10 | thread_num:10 |
| Fcos_efficientnetb0 | 1x3x512x512 | 0.887 | 7227.14 | thread_num:10 |
| Ganet_mixvargenet | 1x3x320x800 | 0.550 | 12439.15 | thread_num:12 |
| Deformable_detr_resnet50 | 1x3x800x1333 | 77.201 | 29.28 | thread_num:8 |
| Stereonetplus_mixvargenet | 2x3x544x960 | 2.022 | 2147.39 | thread_num:8 |
| Centerpoint_mixvargnet_multitask | points: 300000x5 voxel_feature: 1x5x20x40000 coors: 40000x4 | 6.263 | 1638.08 | thread_num:18 |
| Unet_mobilenetv1 | 1x3x1024x2048 | 0.881 | 6401.99 | thread_num:10 |
| Densetnt_vectornet | goals_2d: 30x1x2048x2 goals_2d_mask: 30x1x2048x1 instance_mask: 30x1x96x1 lane_feat: 30x9x64x11 traj_feat: 30x19x32x9 | 3.560 | 1108.38 | thread_num:12 |
| Maptroe_henet_tinym_bevformer | img: 6x3x480x800 osm_mask: 1x1x50x100 queries_rebatch_grid: 6x20x100x2 restore_bev_grid: 1x100x100x2 reference_points_rebatch: 6x2000x4x2 bev_pillar_counts: 1x5000x1 | 6.439 | 625.40 | thread_num:8 |
| Qcnet_oe | valid_mask: 1x30x10 valid_mask_a2a: 1x10x30x30 agent_type: 1x30x1 x_a_cur: 1x1x30x1,1x1x30x1,1x1x30x1,1x1x30x1 r_pl2a_cur: 1x1x30x80,1x1x30x80,1x1x30x80 r_t_cur: 1x1x30x6,1x1x30x6,1x1x30x6,1x1x30x6 r_a2a_cur: 1x1x30x30,1x1x30x30,1x1x30x30 x_a_mid_emb: 1x30x2x128 x_a: 1x30x6x128 pl_type,is_intersection: 1x80 r_pl2pl: 1x1x80x80,1x1x80x80,1x1x80x80 r_pt2pl: 1x1x80x50,1x1x80x50,1x1x80x50 mask_pl2pl: 1x80x80 magnitude,pt_type,side,mask: 1x80x50 mask_a2m: 1x30x30 mask_dst: 1x30x1 type_pl2pl: 1x80x80 | 2.611 | 1723.45 | thread_num:8 |