English

AI Benchmark User Guide

AI Benchmark Sample Package contains the most frequently used performance and accuracy evaluation samples of classification, detection, segmentation, optical-flow, tracking estimation, lidar multitask, bev, depth estimation and online map construction models. The performance evaluation samples include single-frame latency evaluation and multithreading evaluation samples, which make full use of the speed of invoking the BPU for evaluation. The pre-build source code, executable programs and evaluation scripts in the AI Benchmark Sample Package allow you to experience the samples, and develop their own applications, which makes development easier.

Release Description

The AI Benchmark sample package is located in the samples/ucp_tutorial/dnn/ai_benchmark/ path of the release package and consists the following main contents:

No.	Name	Description
1	code	This folder contains sample source code and compilation scripts.
2	runtime	Dev board operating environment of the AI Benchmark Sample Package.

Sample Code Package

Directory of the sample code package is shown as below:

├── code                                   # sample source code
│   ├── build_ptq_runtime.sh
│   ├── build_qat_runtime.sh
│   ├── build.sh                           # common build script
│   ├── CMakeLists.txt
│   ├── include                            # source code header files
│   │   ├── base
│   │   ├── input
│   │   ├── method
│   │   ├── output
│   │   ├── plugin
│   │   └── utils
│   ├── README.md
│   ├── resolve.sh                         # download script for model performance evaluation dataset
│   └── src                                # sample source code
│       ├── input
│       ├── method
│       ├── output
│       ├── plugin
│       ├── simple_example.cc
│       └── utils
├── runtime                                # Sample package runtime environment
│   ├── ptq                                # PTQ (post-training quantization) model samples
│   │   ├── data                           # Model performance evaluation dataset
│   │   ├── mini_data                      # performance evaluations dataset
│   │   ├── model                          # PTQ (post-training quantization) solution model
│   │   │   ├── README.md
│   │   │   └── runtime -> ../../../../../../model_zoo/runtime/nash-e/ai_benchmark/ptq   # Soft link to the model in the OE package, the board environment you need to specify the model path
│   │   ├── README.md
│   │   ├── script                         # execution script
│   │   │   ├── aarch64                    # executable files generated by the compilation and dependencies
│   │   │   ├── classification             # samples classification models
│   │   │   ├── config                     # configuration files of model inference 
│   │   │   ├── detection                  # detection model samples
│   │   │   ├── segmentation               # segmentation model samples
│   │   │   ├── env.sh                     # basic environment script
│   │   │   └── README.md
│   │   └── tools                          # accuracy evaluation tools
│   └── qat                                # QAT training model samples
│       ├── data                           # Model performance evaluation dataset
│       ├── mini_data                      # model performance evaluations dataset
│       ├── model                          # QAT scheme model
│       │   ├── README.md
│       │   └── runtime -> ../../../../../../model_zoo/runtime/nash-e/ai_benchmark/qat   # Soft link to the model in the OE package, the board environment you need to specify the model path
│       ├── README.md
│       ├── script                        # execution script
│       │   ├── aarch64                   # executable files generated by the compilation and dependencies
│       │   ├── bev                       # bev model samples
│       │   ├── classification            # samples classification models
│       │   ├── config                    # model inference profile
│       │   ├── detection                 # detection model samples
│       │   ├── disparity_pred            # disparity pred model samples
│       │   ├── multitask                 # multi task model samples
│       │   ├── online_map                # online map construction model samples
│       │   ├── opticalflow               # optical flow model samples
│       │   ├── segmentation              # segmentation model samples
│       │   ├── tracking                  # tracking model samples
│       │   ├── traj_pred                 # traj pred model samples
│       │   ├── env.sh                    # basic environment scripts
│       │   └── README.md
│       └── tools                         # pre-processing and accuracy evaluation tools
└── README.md

The code directory contains the source code of the evaluation program, used to evaluate model performance and accuracy.
The runtime directory contains various pre-compiled application programs and evaluation scripts, used to evaluate the accuracy and performance of different models in Horizon's BPU (Brain Processing Unit).

Sample Models

Model releases for the AI Benchmark sample package include PTQ model and QAT model releases:

You can get the model_zoo of the PTQ model by executing the script resolve_ai_benchmark_ptq.sh of the samples/ai_toolchain/model_zoo/runtime/${march}/ai_benchmark/ path. The ${march} in the path can be set to nash-e or nash-p , corresponding respectively to the S100/S100P and S600 platforms. You can choose the appropriate one based on your needs.
You can get the model_zoo of the QAT model by executing the script resolve_ai_benchmark_qat.sh of the samples/ai_toolchain/model_zoo/runtime/${march}/ai_benchmark/ path. The ${march} in the path can be set to nash-e or nash-p , corresponding respectively to the S100/S100P and S600 platforms. You can choose the appropriate one based on your needs.
The ptq/model folder contains the model files, and the runtime folder is a symbolic link that by default points to ../../../../../../model_zoo/runtime/nash-e/ai_benchmark/ptq . If you need to run a model with an architecture other than nash-e , you can manually change nash-e in the link path to nash-b or nash-p .
The qat/model folder contains the model files, and the runtime folder is a symbolic link that by default points to ../../../../../../model_zoo/runtime/nash-e/ai_benchmark/qat . If you need to run a model with an architecture other than nash-e , you can manually change nash-e in the link path to nash-b or nash-p .

Among them, which contain the commonly used classification, detection, segmentation and optical flow prediction models, and the naming rules of the models is {model_name}_{backbone}_{input_size}_{input_type}.

Note

Both the PTQ and QAT models in model_zoo are compiled by the original model.

The performance data of the model in the AI Benchmark sample package can be found in the section Benchmark of Model Performance depending on your platform.

Public Datasets

The dataset will be used in the sample, you can download the corresponding dataset in section Dataset Download. If you have any questions during the data preparation process, please contact Horizon.

Environment Configuration

Before using the AI Benchmark sample package, you need to ensure that the development board environment and compilation environment are available:

Prepare the Development Board
1. After getting the development board, upgrade the system image file to the version recommended by the sample package.
2. Make sure the remote connection between local dev machine and the dev board.
Prepare the compilation environment

Install the aarch64-none-linux-gnu-gcc and aarch64-none-linux-gnu-g++ cross-compilation tool in current environment. Then execute the build_ptq_runtime.sh and build_qat_runtime.sh scripts in the code directory to compile the executable program in the real machine environment with one click. The executable programs and corresponding dependencies will be copied into the aarch64 sub-folders of the runtime/ptq/script and runtime/qat/script folders automatically.

Note

The cross-compilation tool specified by the build.sh script is located in the /opt folder. If you want to install it into some other locations, please modify the script.

export CC=/arm-gnu-toolchain-12.2.rel1-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-gcc
export CXX=/arm-gnu-toolchain-12.2.rel1-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-g++

How to Use

Evaluation Scripts

Evaluation sample scripts are in the script and tools folders.

The script folder contains the scripts used for evaluating frequently used classification, detection, segmentation, optical-flow, tracking estimation, lidar multitask, bev and depth estimation models in dev board. There are three scripts under each model:

Script	Description
fps.sh	The script implements FPS statistics (multi-threading scheduling. You can freely specify number of threads as needed) .
latency.sh	The script implements statistics of single-frame latency (one thread, single-frame).
accuracy.sh	The script is used for evaluating model accuracy.

script:

├── aarch64                                    # executable files generated by the compilation and dependencies
│   ├── bin
│   └── lib
├── env.sh                                     # base config
├── config                                     
│   ├── model                                  
│   │   ├── data_name_list                     # image name config
│   │   └── input_init                         # model input config
│   │   └── input_order                         # model input order
│   ├── preprocess
│   │   └── centerpoint_preprocess_5dim.json   # preprocess config
│   └── reference_points                       # reference points config
│   │   ├── bev_gkt_mixvargenet_multitask_nuscenes  
│   │   └── ...
├── detection                                  # detection models
│   ├── fcos_efficientnetb0_mscoco             # there are other models in this directory, just use this model directory as a reference
│   │   ├── accuracy.sh
│   │   ├── fps.sh
│   │   ├── latency.sh
│   │   ├── workflow_accuracy.json             # accuracy config file
│   │   ├── workflow_fps.json                  # fps config file
│   │   └── workflow_latency.json              # latency config file
│   └──...
├── ...
└── README.md

The (PTQ)tools folder contains the precision calculation scripts under python_tools, which used for accuracy evaluation.

python_tools
└── accuracy_tools
    ├── cls_imagenet_eval.py
    ├── det_coco_eval.py
    ├── det_voc_eval.py
    ├── metrics
    │   ├── cityscapes_metric.py
    │   ├── coco_config.py
    │   ├── coco_metric.py
    │   └── voc_metric.py
    └── seg_cityscapes_eval.py

Attention

Run the following commands before the evaluation and copy the ptq (or the qat) directory to the dev board.

scp -r samples/ucp_tutorial/dnn/ai_benchmark/runtime/ptq root@192.168.1.1:/userdata/ptq/
scp -r samples/ucp_tutorial/dnn/ai_benchmark/runtime/qat root@192.168.1.1:/userdata/qat/

Parameters of json Configuration File

This section we provide you the brief introduction of the configuration for the workflow_fps.json, workflow_latency.json and workflow_accuracy.json. The configuration can be simply divided into input_config, output_config and the workflow configuration.

Note

The configuration parameters given below are the general configuration, some sample models will have additional configuration due to the model specificity, please refer to the sample model json file for details.

input_config

Parameter Name	Description	Involved json Files
input_type	Specify the input data format, support image or bin file.	fps.json, latency.json, accuracy.json
height	Specify the input data height.	fps.json, latency.json, accuracy.json
width	Specify the input data width.	fps.json, latency.json, accuracy.json
data_type	Specify the input data type. Supported type can refer to `hbDNNDataType`section. The corresponding data types are sorted from top to bottom, corresponding to the numbers 0,1,2... For example, HB_DNN_TENSOR_TYPE_S4 corresponds to the 0, HB_DNN_TENSOR_TYPE_U4 corresponds to the 1, HB_DNN_TENSOR_TYPE_S8 corresponds to the 2...	fps.json, latency.json, accuracy.json
image_list_file	Specify the path of the lst file of the preprocessing dataset.	fps.json, latency.json, accuracy.json
need_pre_load	Specify whether to read the dataset using the preload method.	fps.json, latency.json, accuracy.json
limit	Specify the threshold for the difference between the amount of input data being processed and has been processed, which is used to control the processing threads for the input data.	fps.json, latency.json, accuracy.json
need_loop	Specify whether to use cyclic read data for evaluation.	fps.json, latency.json, accuracy.json
max_cache	When this parameter takes effect, the image will be pre-processed and read into memory. To ensure the stable running of your application, do not set too large a value, we recommend that you set a value of no more than 30.	fps.json, latency.json, accuracy.json

output_config

Parameter Name	Description	Involved json Files
output_type	Specify the output data type.	fps.json, latency.json and accuracy.json
in_order	Specify whether to output in order.	fps.json, latency.json and accuracy.json
enable_view_output	Specify whether to visualize the output.	fps.json and latency.json
image_list_enable	When visualizing, set to true to save the output as the image type.	fps.json and latency.json
view_output_dir	Specify the path of the visualization result output file.	fps.json and latency.json
eval_enable	Specify whether to evaluate the accuracy.	accuracy.json
output_file	Specify the model output result file.	accuracy.json

Workflow Configuration

Model inference configurations:

Parameter Name	Description	Involved json Files
method_type	Specify the model inference method, which here needs to be configured as `InferMethod`.	fps.json, latency.json, accuracy.json
method_config	Specify the model inference parameters. core: specify the inference core id. model_file: specify the model file.	fps.json, latency.json, accuracy.json

Post-processing configurations:

Parameter Name	Description	Involved json Files
thread_count	Specify the post-processing thread count, in range `1-8`.	fps.json, latency.json and accuracy.json
method_type	Specify the post-processing method.	fps.json, latency.json and accuracy.json
method_config	Specify the post-processing parameters.	fps.json, latency.json and accuracy.json

Performance Evaluation

Performance evaluation is divided into latency and fps.

How to Use Performance Evaluation Scripts

latency:

In the directory of the to-be-evaluated model, run sh latency.sh to evaluate single frame latency, as shown below:

I0419 02:35:07.041095 39124 output_plugin.cc:80]  Infer latency:  [avg:  13.124ms,  max:  13.946ms,  min:  13.048ms], Post process latency: [avg:  3.584ms,  max:  3.650ms,  min:  3.498ms].

Note

infer denotes the time consumption of model inference.
Post process denotes the time consumption of post-processing.

fps:

This function uses multi-threaded concurrency and is designed to allow the model to reach the ultimate performance on BPU. Due to the multi-thread concurrency and data sampling, the frame rate value will be low during the start-up phase, then the frame rate will increase and gradually stabilize, with the frame rate fluctuating within 0.5%. To test the frame rate, go to the model directory and run sh fps.sh, as shown below.

I0419 02:35:00.044417 39094 output_plugin.cc:109]  Throughput: 1129.39fps      # 模型帧率

About Command-line Parameters

The fps.sh script is shown as below:

#!/bin/sh

source ../../env.sh
export SHOW_FPS_LOG=1
export STAT_CYCLE=100                           # specify environment variable, fps log printing frequency

${app} \
  --config_file=workflow_fps.json \
  --log_level=1

The latency.sh script is shown as below:

#!/bin/sh

source ../../env.sh
export SHOW_LATENCY_LOG=1                         # specify environment variable, print latency level log
export STAT_CYCLE=50                              # specify environment variable, latency log printing frequency

${app} \
  --config_file=workflow_latency.json \
  --log_level=1

Result Visualization

If you want to see the effect of a single inference of the model, you can modify workflow_latency.json and re-run the latency.sh script to generate the display effect in the output_dir directory.

Attention

When the display effect is generated, the script will run slowly due to the dump effect. Only the latency.sh script dump is supported.

The Visual operation steps are as follows:

Modify the latency configuration file

"output_config": { "output_type": "image", "enable_view_output": true, # turn on visualization "view_output_dir": "./output_dir", # visualization result output path "image_list_enable": true, "in_order": false }
Execute the latency.sh script

sh latency.sh

Attention

The visualization of the bev model needs to specify the scene information and the path of the homography matrix. The homography matrix is used for the conversion of the camera perspective and the bird's-eye view. Different scenes have their own homography matrices. The visualization of the online map construction model needs to specify the perception range of local map.

The workflow_latency.json configuration file of the bev model is recommended to be modified as follows:

"output_config": {
  "output_type": "image",
  "enable_view_output": true,                            # turn on visualization
  "view_output_dir": "./output_dir",                     # visualization result output path
  "bev_ego2img_info": [
    "../../config/visible/bev/scenes.json",              # scene information of input file           
    "../../config/visible/bev/boston.bin",               # homography matrix of the boston scene
    "../../config/visible/bev/singapore.bin"             # homography matrix of the singapore scene
  ],
  "image_list_enable": true,
  "in_order": false
}

The workflow_latency.json configuration file of the online map construction model is recommended to be modified as follows:

"output_config": {
  "output_type": "image",
  "enable_view_output": true,                            # turn on visualization
  "view_output_dir": "./output_dir",                     # visualization result output path
  "bev_range": [
    -30.0,          
    -15.0,
    -10.0,
    30.0,
    15.0,
    10.0
  ],                                                     # perception range of local map
  "image_list_enable": true,
  "in_order": false
}

Attention

QCNet integrates model visualization and accuracy evaluation.

The visualization method of qcnet is as follows:

python3 tools/bpu_validation.py
            -c                 qcnet_oe_argoverse2.py   # Config file corresponding to QCNet model
            --result-path      eval.log                 # Path for inference results (visualization storage path)
            --gt-files-path    argoverse2_qcnet/val     # Mount path of Argoverse2 validation dataset (ground truth files)

The visualization results that can be achieved by different types of models are different, refer to the table below:

Model Category	Visualization
classification
detection 2d
detection 3d
segmentation
keypoint
lane line
optical flow
lidar
lidar multitask
bev
raj_pred
disparity_pred
online map construction
occ_pred

Attention

If you need to visualize images other than minidata during trajectory prediction visualization, you need to configure additional road information and trajectory information files in minidata/argoverse1/visualization. You can use the densent_process.py preprocessing script to generate configuration files, and set --is-gen-visual-config parameter to true.

Model Accuracy Evaluation

Take the following 5 steps to perform the model evaluation:

Data pre-process.
Data mounting.
The lst file generation.
Model inference.
Model accuracy computing.

Data Pre-processing

The following section will provide the description of the PTQ and QAT model data preprocessing methods.

PTQ Model Data Pre-processing:

To the PTQ model data pre-processing, run the hb_eval_preprocess tool in x86 to pre-process data. The so-called pre-processing refers to the special processing operations before images are fed into the model. For example: resize, crop and padding, etc. The tool is integrated into the horizon_tc_ui tool and it will be available after the tool is installed using the install script. After the raw dataset is pre-processed by the tool, the corresponding pre-processed binary file .bin file set of the model will be generated.

Tip

About the hb_eval_preprocess tool command line parameters and usage, you can type hb_eval_preprocess -h, or see the hb_eval_preprocess Tool in the PTQ tools guide.

The datasets corresponding to each model in the sample package are described in detail below, as well as the pre-processing operations for the corresponding datasets.

The datasets used for PTQ models include the following:

Dataset Description

Dataset	Description
VOC Dataset	For evaluation of detection models using the `VOC2012` dataset. The dataset download and reference download structure can be found in section Preparing Datasets. The sample mainly use the val.txt file under the Main folder, the source images under the JPEGImages folder and the annotations under the Annotations folder. 2. Dataset preprocessing methods example: `hb_eval_preprocess -m ssd_mobilenetv1 \ -i VOCdevkit/VOC2012/JPEGImages \ -v VOCdevkit/VOC2012/ImageSets/Main/val.txt \ -o ./pre_ssd_mobilenetv1`
COCO Dataset	For evaluation of detection models using the `coco_val2017` dataset. 1. The dataset download and reference download structure can be found in section Preparing Datasets. The samples mainly use the instances_val2017.json annotation file under the annotations folder images under the images folder. 2. Dataset preprocessing methods example: `hb_eval_preprocess -m model_name \ -i coco/coco_val2017/images \ -o ./pre_model_name`
ImageNet Dataset	For evaluation of classification models using the `imagenet` dataset. 1. The dataset download and reference download structure can be found in section Preparing Datasets. The samples mainly use the annotation file val.txt and the source images in the val directory. 2. Dataset preprocessing methods example: `hb_eval_preprocess -m model_name \ -i imagenet/val \ -o ./pre_model_name`
Cityscapes Dataset	For evaluation of segmentation models using the `cityscapes` dataset. 1. The dataset download and reference download structure can be found in section Preparing Datasets. The samples mainly use the annotation files under the ./gtFine/val folder and the source images in the ./leftImg8bit/val folder. 2. Dataset preprocessing methods example: `hb_eval_preprocess -m model_name \ -i cityscapes/leftImg8bit/val \ -o ./pre_model_name`

VOC

Dataset

For evaluation of detection models using the VOC2012 dataset.

The dataset download and reference download structure can be found in section Preparing Datasets. The sample mainly use the val.txt file under the Main folder, the source images under the JPEGImages folder and the annotations under the Annotations folder.

2. Dataset preprocessing methods example:

hb_eval_preprocess -m ssd_mobilenetv1 \
                   -i VOCdevkit/VOC2012/JPEGImages \
                   -v VOCdevkit/VOC2012/ImageSets/Main/val.txt \
                   -o ./pre_ssd_mobilenetv1

COCO

Dataset

For evaluation of detection models using the coco_val2017 dataset.

1. The dataset download and reference download structure can be found in section Preparing Datasets. The samples mainly use the instances_val2017.json annotation file under the annotations folder images under the images folder.

2. Dataset preprocessing methods example:

hb_eval_preprocess -m model_name \
                   -i coco/coco_val2017/images \
                   -o ./pre_model_name

ImageNet

Dataset

For evaluation of classification models using the imagenet dataset.

1. The dataset download and reference download structure can be found in section Preparing Datasets. The samples mainly use the annotation file val.txt and the source images in the val directory.

2. Dataset preprocessing methods example:

hb_eval_preprocess -m model_name \
                   -i imagenet/val \
                   -o ./pre_model_name

Cityscapes

Dataset

For evaluation of segmentation models using the cityscapes dataset.

1. The dataset download and reference download structure can be found in section Preparing Datasets. The samples mainly use the annotation files under the ./gtFine/val folder and the source images in the ./leftImg8bit/val folder.

2. Dataset preprocessing methods example:

hb_eval_preprocess -m model_name \
                   -i cityscapes/leftImg8bit/val \
                   -o ./pre_model_name

QAT Model Data Pre-processing:

HAT provides the tools to preprocess the LMDB dataset. The datasets corresponding to each model are described in detail as below, as well as the pre-processing operations for the corresponding datasets.

Tip

Location of the data preprocessing scripts: oe package/samples/ai_toolchain/horizon_model_train_sample/scripts. The data preprocessing scripts need to be run in a GPU environment. Before use, please package the original dataset into LMDB data，then mount it to the tmp_data directory in the working path.

Dataset Description

Dataset	Description
ImageNet Dataset	The ImageNet dataset is used for evaluating the mixvargenet_imagenet, mobilenetv1_imagenet, mobilenetv2_imagenet, resnet18_imagenet, resnet50_imagenet, horizon_swin_transformer_imagenet, vit_small_imagenet and vargnetv2_imagenet QAT models. Dataset preprocessing methods: 1. After running the script, an `imagenet_gt.jsonl` file will be generated under the specified save-path for precision calculation. 2. The data preprocessing logic for these models is consistent, so the data generated after running the script once can be reused for other models. mixvargenet_imagenet `python3 tools/bpu_val_preprocess.py -c configs/classification/mixvargenet_imagenet.py \ --save-path ./data/imagenet/` mobilenetv1_imagenet `python3 tools/bpu_val_preprocess.py -c configs/classification/mobilenetv1_imagenet.py \ --save-path ./data/imagenet/` mobilenetv2_imagenet `python3 tools/bpu_val_preprocess.py -c configs/classification/mobilenetv2_imagenet.py \ --save-path ./data/imagenet/` resnet18_imagenet `python3 tools/bpu_val_preprocess.py -c configs/classification/resnet18_imagenet.py \ --save-path ./data/imagenet/` resnet50_imagenet `python3 tools/bpu_val_preprocess.py -c configs/classification/resnet50_imagenet.py \ --save-path ./data/imagenet/` horizon_swin_transformer_imagenet `python3 tools/bpu_val_preprocess.py -c configs/classification/horizon_swin_transformer_imagenet.py \ --save-path ./data/imagenet/` vit_small_imagenet `python3 tools/bpu_val_preprocess.py -c configs/classification/vit_small_imagenet.py \ --save-path ./data/imagenet/` vargnetv2_imagenet `python3 tools/bpu_val_preprocess.py -c configs/classification/vargnetv2_imagenet.py \ --save-path ./data/imagenet/`
COCO Dataset	The COCO dataset is used for evaluating the fcos_efficientnetb0_mscoco, detr_resnet50_mscoco, detr_efficientnetb3_mscoco and deform_detr_resnet50_mscoco QAT models. Dataset preprocessing methods: 1. After running the script, a `meta.jsonl` file will be generated under the specified save-path for precision calculation. 2. The data preprocessing logic for detr_resnet50_mscoco and detr_efficientnetb3_mscoco are consistent, so the data generated after running the script once can be reused for the another models. fcos_efficientnetb0_mscoco `python3 tools/bpu_val_preprocess.py -c configs/detection/fcos/fcos_efficientnetb0_mscoco.py \ --save-path ./data/coco/` detr_resnet50_mscoco `python3 tools/bpu_val_preprocess.py -c configs/detection/detr/detr_resnet50_mscoco.py \ --save-path ./data/coco/` detr_efficientnetb3_mscoco `python3 tools/bpu_val_preprocess.py -c configs/detection/detr/detr_efficientnetb3_mscoco.py \ --save-path ./data/coco/` deform_detr_resnet50_mscoco `python3 tools/bpu_val_preprocess.py -c configs/detection/deform_detr/deform_detr_resnet50_mscoco.py \ --save-path ./data/coco/`
Cityscapes Dataset	The Cityscapes dataset is used for the evaluation of the QAT segmentation model unet_mobilenetv1_cityscapes. Dataset preprocessing methods: After running the script, a directory named `gts` will be generated under the specified save-path for precision calculation. `python3 tools/bpu_val_preprocess.py -c configs/segmentation/unet_mobilenetv1_cityscapes.py \ --save-path ./data/cityscapes/`
Kitti3D Dataset	The Kitti3D dataset is used for evaluating the pointpillars_kitti_car model. Dataset preprocessing methods: After running the script, a `val_gt_infos.pkl` file will be generated under the specified save-path for precision calculation. `python3 tools/bpu_val_preprocess.py -c configs/detection/pointpillars/pointpillars_kitti_car.py \ --save-path ./data/kitti3d/`
Culane Dataset	The Culane dataset is used for evaluating the ganet_mixvargenet_culane model. Dataset preprocessing methods: After running the script, a `gt.json` file will be generated under the specified save-path for precision calculation. `python3 tools/bpu_val_preprocess.py -c configs/lane_pred/ganet/ganet_mixvargenet_culane.py \ --save-path ./data/culane/`
Nuscenes Dataset	The Nuscenes dataset is used for evaluating the fcos3d_efficientnetb0_nuscenes, centerpoint_pointpillar_nuscenes, centerpoint_mixvargnet_multitask_nuscenes, bev_gkt_mixvargenet_multitask_nuscenes, bev_lss_efficientnetb0_multitask_nuscenes, bev_ipm_efficientnetb0_multitask_nuscenes, bev_ipm_4d_efficientnetb0_multitask_nuscenes, bevformer_tiny_resnet50_detection_nuscenes, maptroe_henet_tinym_bevformer_nuscenes, detr3d_efficientnetb3_nuscenes, petr_efficientnetb3_nuscenes and flashocc_henet_lss_occ3d_nuscenes models. Dataset preprocessing methods: 1. For the following models: flashocc_henet_lss_occ3d_nuscenes, bevformer_tiny_resnet50_detection_nuscenes, bev_lss_efficientnetb0_multitask_nuscenes, petr_efficientnetb3_nuscenes, detr3d_efficientnetb3_nuscenes, bev_gkt_mixvargenet_multitask_nuscenes, bev_ipm_4d_efficientnetb0_multitask_nuscenes, bev_ipm_efficientnetb0_multitask_nuscenes and centerpoint_mixvargnet_multitask_nuscenes, running the script will generate a `val_gt_infos.pkl` file under the specified save-path for precision calculation. 2. For the model maptroe_henet_tinym_bevformer_nuscenes, running the script will generate a `nuscenes_map_anns_val.json` file under the specified save-path for precision calculation. 3. According to the README in the reference model algorithm package, download the corresponding model's reference points and place them under `./tmp_data/nuscenes/reference_points`, organized as follows: `├─── tmp_data ├─── ├─── nuscenes ├─── ├─── ├─── reference_points ├─── ├─── ├─── ├─── bevformer_tiny_resnet50_detection_nuscenes` centerpoint_pointpillar_nuscenes `python3 tools/bpu_val_preprocess.py \ -c configs/detection/centerpoint/centerpoint_pointpillar_nuscenes.py \ --save-path ./data/nuscenes/` flashocc_henet_lss_occ3d_nuscenes `python3 tools/bpu_val_preprocess.py \ -c configs/occ/flashocc_henet_lss_occ3d_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "flashocc_henet_lss_occ3d_nuscenes"` bevformer_tiny_resnet50_detection_nuscenes `python3 tools/bpu_val_preprocess.py \ -c configs/bev/bevformer_tiny_resnet50_detection_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "bevformer_tiny_resnet50_detection_nuscenes"` bev_lss_efficientnetb0_multitask_nuscenes `python3 tools/bpu_val_preprocess.py \ -c configs/bev/bev_lss_efficientnetb0_multitask_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "bev_lss_efficientnetb0_multitask_nuscenes"` petr_efficientnetb3_nuscenes `python3 tools/bpu_val_preprocess.py \ -c configs/bev/petr_efficientnetb3_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "petr_efficientnetb3_nuscenes"` detr3d_efficientnetb3_nuscenes `python3 tools/bpu_val_preprocess.py \ -c configs/bev/detr3d_efficientnetb3_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "detr3d_efficientnetb3_nuscenes"` bev_gkt_mixvargenet_multitask_nuscenes `python3 tools/bpu_val_preprocess.py \ -c configs/bev/bev_gkt_mixvargenet_multitask_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "bev_gkt_mixvargenet_multitask_nuscenes"` bev_ipm_4d_efficientnetb0_multitask_nuscenes `python3 tools/bpu_val_preprocess.py \ -c configs/bev/bev_ipm_4d_efficientnetb0_multitask_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "bev_ipm_4d_efficientnetb0_multitask_nuscenes"` bev_ipm_efficientnetb0_multitask_nuscenes `python3 tools/bpu_val_preprocess.py \ -c configs/bev/bev_ipm_efficientnetb0_multitask_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "bev_ipm_efficientnetb0_multitask_nuscenes"` fcos3d_efficientnetb0_nuscenes `python3 tools/bpu_val_preprocess.py \ -c configs/detection/fcos3d/fcos3d_efficientnetb0_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args \ --file_path_txt ucp_tutorial/dnn/ai_benchmark/runtime/qat/script/config/model/data_name_list/nuscenes_names.txt` centerpoint_mixvargnet_multitask_nuscenes `python3 tools/bpu_val_preprocess.py \ -c configs/lidar_multi_task/centerpoint_mixvargnet_multitask_nuscenes.py \ --save-path ./data/nuscenes/` maptroe_henet_tinym_bevformer_nuscenes `python3 tools/bpu_val_preprocess.py \ -c configs/map/maptroe_henet_tinym_bevformer_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --model maptroe_henet_tinym_bevformer_nuscenes \ --reference_path ucp_tutorial/dnn/ai_benchmark/runtime/qat/script/config/reference_points`
Mot17 Dataset	The Mot17 dataset is used for evaluating the motr_efficientnetb3_mot17 model. Dataset preprocessing methods: `python3 tools/bpu_val_preprocess.py -c configs/track_pred/motr_efficientnetb3_mot17.py \ --save-path ./data/mot17/`
Carfusion Dataset	The Carfusion dataset is used for evaluating the keypoint_efficientnetb0_carfusion model. Dataset preprocessing methods: After running the script, a `gt.json` file will be generated under the specified save-path for precision calculation. `python3 tools/bpu_val_preprocess.py -c configs/keypoint/keypoint_efficientnetb0_carfusion.py \ --save-path ./data/carfusion/`
Argoverse1 Dataset	The Argoverse1 dataset is used for evaluating the densetnt_argoverse1 model. Dataset preprocessing methods: After running the script, a `meta` directory will be generated under the specified save-path for precision calculation. `python3 tools/bpu_val_preprocess.py -c configs/traj_pred/densetnt_vectornet_argoverse1.py \ --save-path ./data/argoverse1/`
Argoverse2 Dataset	The Argoverse2 dataset is used for evaluating the qcnet_oe_argoverse2 model. Dataset preprocessing methods: After running the script, a `val` directory will be generated under the specified save-path for precision calculation. `python3 tools/bpu_val_preprocess.py -c configs/traj_pred/qcnet_oe_argoverse2.py \ --save-path ./data/argoverse2/`
SceneFlow Dataset	The SceneFlow dataset is used for evaluating the stereonetplus_mixvargenet_sceneflow model. Dataset preprocessing methods: After running the script, a `val_gt_infos.pkl` file will be generated under the specified save-path for precision calculation. `python3 tools/bpu_val_preprocess.py -c configs/disparity_pred/stereonet/stereonetplus_mixvargenet_sceneflow.py \ --save-path ./data/sceneflow/`

ImageNet

Dataset

The ImageNet dataset is used for evaluating the mixvargenet_imagenet, mobilenetv1_imagenet, mobilenetv2_imagenet,

resnet18_imagenet, resnet50_imagenet,

horizon_swin_transformer_imagenet, vit_small_imagenet and vargnetv2_imagenet QAT models.

Dataset preprocessing methods:

1. After running the script, an imagenet_gt.jsonl file will be generated under

the specified save-path for precision calculation.

2. The data preprocessing logic for these models is consistent,

so the data generated after running the script once can be reused for other models.

mixvargenet_imagenet
python3 tools/bpu_val_preprocess.py -c configs/classification/mixvargenet_imagenet.py \ --save-path ./data/imagenet/
mobilenetv1_imagenet
python3 tools/bpu_val_preprocess.py -c configs/classification/mobilenetv1_imagenet.py \ --save-path ./data/imagenet/
mobilenetv2_imagenet
python3 tools/bpu_val_preprocess.py -c configs/classification/mobilenetv2_imagenet.py \ --save-path ./data/imagenet/
resnet18_imagenet
python3 tools/bpu_val_preprocess.py -c configs/classification/resnet18_imagenet.py \ --save-path ./data/imagenet/
resnet50_imagenet
python3 tools/bpu_val_preprocess.py -c configs/classification/resnet50_imagenet.py \ --save-path ./data/imagenet/
horizon_swin_transformer_imagenet
python3 tools/bpu_val_preprocess.py -c configs/classification/horizon_swin_transformer_imagenet.py \ --save-path ./data/imagenet/
vit_small_imagenet
python3 tools/bpu_val_preprocess.py -c configs/classification/vit_small_imagenet.py \ --save-path ./data/imagenet/
vargnetv2_imagenet
python3 tools/bpu_val_preprocess.py -c configs/classification/vargnetv2_imagenet.py \ --save-path ./data/imagenet/

COCO

Dataset

The COCO dataset is used for evaluating the fcos_efficientnetb0_mscoco, detr_resnet50_mscoco,

detr_efficientnetb3_mscoco and deform_detr_resnet50_mscoco QAT models.

Dataset preprocessing methods:

1. After running the script, a meta.jsonl file will be generated under

the specified save-path for precision calculation.

2. The data preprocessing logic for detr_resnet50_mscoco and detr_efficientnetb3_mscoco are consistent,

so the data generated after running the script once can be reused for the another models.

fcos_efficientnetb0_mscoco
python3 tools/bpu_val_preprocess.py -c configs/detection/fcos/fcos_efficientnetb0_mscoco.py \ --save-path ./data/coco/
detr_resnet50_mscoco
python3 tools/bpu_val_preprocess.py -c configs/detection/detr/detr_resnet50_mscoco.py \ --save-path ./data/coco/
detr_efficientnetb3_mscoco
python3 tools/bpu_val_preprocess.py -c configs/detection/detr/detr_efficientnetb3_mscoco.py \ --save-path ./data/coco/
deform_detr_resnet50_mscoco
python3 tools/bpu_val_preprocess.py -c configs/detection/deform_detr/deform_detr_resnet50_mscoco.py \ --save-path ./data/coco/

Cityscapes

Dataset

The Cityscapes dataset is used for the evaluation of the QAT segmentation model unet_mobilenetv1_cityscapes.

Dataset preprocessing methods:

After running the script, a directory named gts will be generated under the specified save-path for precision calculation.

python3 tools/bpu_val_preprocess.py -c configs/segmentation/unet_mobilenetv1_cityscapes.py \
                                                       --save-path ./data/cityscapes/

Kitti3D

Dataset

The Kitti3D dataset is used for evaluating the pointpillars_kitti_car model.

Dataset preprocessing methods:

After running the script, a val_gt_infos.pkl file will be generated under the specified save-path for precision calculation.

python3 tools/bpu_val_preprocess.py -c configs/detection/pointpillars/pointpillars_kitti_car.py \
                                                       --save-path ./data/kitti3d/

Culane

Dataset

The Culane dataset is used for evaluating the ganet_mixvargenet_culane model.

Dataset preprocessing methods:

After running the script, a gt.json file will be generated under the specified save-path for precision calculation.

python3 tools/bpu_val_preprocess.py -c configs/lane_pred/ganet/ganet_mixvargenet_culane.py \
                                                       --save-path ./data/culane/

Nuscenes

Dataset

The Nuscenes dataset is used for evaluating the fcos3d_efficientnetb0_nuscenes, centerpoint_pointpillar_nuscenes,

centerpoint_mixvargnet_multitask_nuscenes, bev_gkt_mixvargenet_multitask_nuscenes, bev_lss_efficientnetb0_multitask_nuscenes,

bev_ipm_efficientnetb0_multitask_nuscenes, bev_ipm_4d_efficientnetb0_multitask_nuscenes, bevformer_tiny_resnet50_detection_nuscenes,

maptroe_henet_tinym_bevformer_nuscenes, detr3d_efficientnetb3_nuscenes, petr_efficientnetb3_nuscenes and flashocc_henet_lss_occ3d_nuscenes models.

Dataset preprocessing methods:

1. For the following models: flashocc_henet_lss_occ3d_nuscenes, bevformer_tiny_resnet50_detection_nuscenes,

bev_lss_efficientnetb0_multitask_nuscenes, petr_efficientnetb3_nuscenes, detr3d_efficientnetb3_nuscenes,

bev_gkt_mixvargenet_multitask_nuscenes, bev_ipm_4d_efficientnetb0_multitask_nuscenes, bev_ipm_efficientnetb0_multitask_nuscenes and

centerpoint_mixvargnet_multitask_nuscenes, running the script will generate a val_gt_infos.pkl file under

the specified save-path for precision calculation.

2. For the model maptroe_henet_tinym_bevformer_nuscenes, running the script will generate a nuscenes_map_anns_val.json file under

the specified save-path for precision calculation.

3. According to the README in the reference model algorithm package, download the corresponding model's reference points and place them under

./tmp_data/nuscenes/reference_points, organized as follows:

├─── tmp_data
├─── ├───  nuscenes                                                                    
├─── ├───  ├───    reference_points                                                    
├─── ├───  ├───    ├───          bevformer_tiny_resnet50_detection_nuscenes

centerpoint_pointpillar_nuscenes
python3 tools/bpu_val_preprocess.py \ -c configs/detection/centerpoint/centerpoint_pointpillar_nuscenes.py \ --save-path ./data/nuscenes/
flashocc_henet_lss_occ3d_nuscenes
python3 tools/bpu_val_preprocess.py \ -c configs/occ/flashocc_henet_lss_occ3d_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "flashocc_henet_lss_occ3d_nuscenes"
bevformer_tiny_resnet50_detection_nuscenes
python3 tools/bpu_val_preprocess.py \ -c configs/bev/bevformer_tiny_resnet50_detection_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "bevformer_tiny_resnet50_detection_nuscenes"
bev_lss_efficientnetb0_multitask_nuscenes
python3 tools/bpu_val_preprocess.py \ -c configs/bev/bev_lss_efficientnetb0_multitask_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "bev_lss_efficientnetb0_multitask_nuscenes"
petr_efficientnetb3_nuscenes
python3 tools/bpu_val_preprocess.py \ -c configs/bev/petr_efficientnetb3_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "petr_efficientnetb3_nuscenes"
detr3d_efficientnetb3_nuscenes
python3 tools/bpu_val_preprocess.py \ -c configs/bev/detr3d_efficientnetb3_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "detr3d_efficientnetb3_nuscenes"
bev_gkt_mixvargenet_multitask_nuscenes
python3 tools/bpu_val_preprocess.py \ -c configs/bev/bev_gkt_mixvargenet_multitask_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "bev_gkt_mixvargenet_multitask_nuscenes"
bev_ipm_4d_efficientnetb0_multitask_nuscenes
python3 tools/bpu_val_preprocess.py \ -c configs/bev/bev_ipm_4d_efficientnetb0_multitask_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "bev_ipm_4d_efficientnetb0_multitask_nuscenes"
bev_ipm_efficientnetb0_multitask_nuscenes
python3 tools/bpu_val_preprocess.py \ -c configs/bev/bev_ipm_efficientnetb0_multitask_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --reference_path ./tmp_data/nuscenes/reference_points/ \ --model "bev_ipm_efficientnetb0_multitask_nuscenes"
fcos3d_efficientnetb0_nuscenes
python3 tools/bpu_val_preprocess.py \ -c configs/detection/fcos3d/fcos3d_efficientnetb0_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args \ --file_path_txt ucp_tutorial/dnn/ai_benchmark/runtime/qat/script/config/model/data_name_list/nuscenes_names.txt
centerpoint_mixvargnet_multitask_nuscenes
python3 tools/bpu_val_preprocess.py \ -c configs/lidar_multi_task/centerpoint_mixvargnet_multitask_nuscenes.py \ --save-path ./data/nuscenes/
maptroe_henet_tinym_bevformer_nuscenes
python3 tools/bpu_val_preprocess.py \ -c configs/map/maptroe_henet_tinym_bevformer_nuscenes.py \ --save-path ./data/nuscenes/ \ --extra-args --model maptroe_henet_tinym_bevformer_nuscenes \ --reference_path ucp_tutorial/dnn/ai_benchmark/runtime/qat/script/config/reference_points

Mot17

Dataset

The Mot17 dataset is used for evaluating the motr_efficientnetb3_mot17 model.

Dataset preprocessing methods:

python3 tools/bpu_val_preprocess.py -c configs/track_pred/motr_efficientnetb3_mot17.py \
                                                       --save-path ./data/mot17/

Carfusion

Dataset

The Carfusion dataset is used for evaluating the keypoint_efficientnetb0_carfusion model.

Dataset preprocessing methods:

After running the script, a gt.json file will be generated under the specified save-path for precision calculation.

python3 tools/bpu_val_preprocess.py -c configs/keypoint/keypoint_efficientnetb0_carfusion.py \
                                                       --save-path ./data/carfusion/

Argoverse1

Dataset

The Argoverse1 dataset is used for evaluating the densetnt_argoverse1 model.

Dataset preprocessing methods:

After running the script, a meta directory will be generated under the specified save-path for precision calculation.

python3 tools/bpu_val_preprocess.py -c configs/traj_pred/densetnt_vectornet_argoverse1.py \
                                                       --save-path ./data/argoverse1/

Argoverse2

Dataset

The Argoverse2 dataset is used for evaluating the qcnet_oe_argoverse2 model.

Dataset preprocessing methods:

After running the script, a val directory will be generated under the specified save-path for precision calculation.

python3 tools/bpu_val_preprocess.py -c configs/traj_pred/qcnet_oe_argoverse2.py \
                                                       --save-path ./data/argoverse2/

SceneFlow

Dataset

The SceneFlow dataset is used for evaluating the stereonetplus_mixvargenet_sceneflow model.

Dataset preprocessing methods:

After running the script, a val_gt_infos.pkl file will be generated under the specified save-path for precision calculation.

python3 tools/bpu_val_preprocess.py -c configs/disparity_pred/stereonet/stereonetplus_mixvargenet_sceneflow.py \
                                                       --save-path ./data/sceneflow/

Model Mounting

Because datasets are huge, it is recommended to mount them for dev board to load, rather than to copy them into the dev board, you need to do the following on the server PC terminal and the board terminal:

Server PC terminal:

Attention

Note that the root permission is required on the server pc terminal to perform the following actions.

Edit one line into /etc/exports: /nfs *(insecure,rw,sync,all_squash,anonuid=1000,anongid=1000,no_subtree_check). Wherein, /nfs denotes mounting path of local machine, it can be replaced by the directory you specify.
Run exportfs -a -r to bring /etc/exports into effect.

Board terminal:

Create the directory to be mounted: mkdir -p /mnt.
Mount: mount -t nfs {PC terminal IP}:/nfs /mnt -o nolock.

Mount the /nfs folder at PC terminal to the /mnt folder in dev board. In this way, mount the folder in which contains preprocessed folder to dev board and create a soft link of /data folder in the /ptq or /qat folder (at the same directory level as /script) in dev board.

Generate lst Files

The running process of precision calculation script in the sample is:

According to the value of image_list_file in workflow_accuracy.json, find the lst file of the corresponding dataset.
Load each preprocessing file according to the path information of preprocessing file stored in lst file, and then perform the inference.

Therefore, after generating the preprocessing file, you need to generate the corresponding LST file, and write the path of each preprocessing file into the lst file, which is related to the storage location of the dataset at the board end. Here, we recommend that its storage location shall be the same level as the ./data/dataset_name/pre_model_name folder.

The structure of the PTQ pre-processed dataset is as follows:

|── ptq
|   |── data
|   |   |── cityscapes
|   |   |   |── pre_deeplabv3plus_efficientnetb0
|   |   |   |   |── xxxx.bin                            # pre-processed binary file
|   |   |   |   |── ....
|   |   |   |── pre_deeplabv3plus_efficientnetb0.lst    # lst file: record the path of each preprocessing file
|   |   |   |── ....
|   |   |── ....
|   |── model
|   |   |── ...
|   |── script
|   |   |── ...

The structure of the QAT pre-processed data set is as follows:

|── qat
|   |── data
|   |   |── carfusion
|   |   |   |── pre_keypoints
|   |   |   |   |── xxxx                   # pre-processed data
|   |   |   |   |── ....
|   |   |   |── pre_carfusion.lst          # lst file: record the path of each preprocessing file
|   |   |── ....
|   |── model
|   |   |── ...
|   |── script
|   |   |── ...

The corresponding lst file is generated by reference as follows:

Except for the densetnt_argoverse1, qcnet_oe_argoverse2, bev, motr_efficientnetb3_mot17, stereonetplus_mixvargenet_sceneflow and maptroe_henet_tinym_bevformer_nuscenes models, the reference generation method of the lst files for the other models:

find ../../../data/coco/pre_centernet_resnet101 -name "*bin*" > ../../../data/coco/pre_centernet_resnet101.lst

Note

The parameters after -name need to be adjusted according to the format of the preprocessed dataset, such as bin, png.

The path stored in the generated lst file is a relative path: ../../../data/coco/pre_centernet_resnet101/ , which can match the workflow_accuracy.json default configuration path.

If you need to change the storage location of the preprocessing dataset, you need to ensure that the corresponding lst file can be used by workflow_accuracy.json. Secondly, it is necessary to ensure that the program can read the corresponding preprocessing file according to the path information in lst file.

For the densetnt_argoverse1, qcnet_oe_argoverse2, bev, motr_efficientnetb3_mot17, stereonetplus_mixvargenet_sceneflow and maptroe_henet_tinym_bevformer_nuscenes models, the reference generation method of the lst files:

Model Name	Reference Generation Method and Description of lst File
densetnt_argoverse1	Reference generation method: `sh generate_acc_lst.sh` The path stored in the generated `lst` file is a relative path: `../../../data/argoverse1/densetnt/`, which can match the `workflow_accuracy.json` default configuration path.
qcnet_oe_argoverse2	We provide a dedicated script, `generate_acc_lst.sh`, for generating lst files. The usage is as follows: `sh generate_acc_lst.sh ../../../data/argoverse2_qcnet/fp32_sample/val \ ../../../data/argoverse2_qcnet` This script requires two input parameters: `preprocessed data mount path` and `output path for the generated list`. The resulting lst file will store paths as relative paths: `../../../data/argoverse2_qcnet/`, which can align with the default configuration path of `workflow_accuracy.json`.
Bev maptroe_henet_tinym_bevformer_nuscenes	Take the bev_ipm_efficientnetb0_multitask_nuscenes as an example. This model has two types of input: images and reference points. The input image and reference point of the same frame have the same name. In order to ensure that the input corresponds, you need to add `sort` to sort by name when executing the `find` command, same with maptroe_henet_tinym_bevformer_nuscenes. Reference generation method: `find ../../../data/nuscenes_bev/images -name "bin" \| \ sort > ../../../data/nuscenes_bev/images.lst find ../../../data/nuscenes_bev/reference_points0 -name "bin" \| \ sort > ../../../data/nuscenes_bev/reference_points0.lst` In addition, bev_ipm_4d_efficientnetb0_multitask_nuscenes is a timing model, which requires input order. Therefore, we provide a script `gen_file_list.sh` for generating lst files, the usage is as follows: `sh gen_file_list.sh` The path stored in the generated `lst` file is a relative path: `../../../data/nuscenes_bev/` , which can match the `workflow_accuracy.json` default configuration path. If you need to change the storage location of the preprocessing dataset, you need to ensure that the corresponding `lst` file can be used by `workflow_accuracy.json`. Secondly, it is necessary to ensure that the program can read the corresponding preprocessing file according to the path information in `lst` file.
motr_efficientnetb3_mot17	Reference generation method: `sh generate_acc_lst.sh` The path stored in the generated `lst` file is a relative path: `../../../data/mot17/motr/` , which can match the `workflow_accuracy.json` default configuration path. If you need to change the storage location of the preprocessing dataset, you need to ensure that the corresponding `lst` file can be used by `workflow_accuracy.json`. Secondly, it is necessary to ensure that the program can read the corresponding preprocessing file according to the path information in `lst` file.
stereonetplus_mixvargenet_sceneflow	Take the stereonetplus_mixvargenet_sceneflow as an example. The input left image and right the same frame have the same name. In order to ensure that the input corresponds, you need to add `sort` to sort by name when executing the `find` command. `find ../../../data/sceneflow/left -name "png" \| \ sort > ../../../data/sceneflow/left.lst find ../../../data/sceneflow/right -name "png" \| \ sort > ../../../data/sceneflow/right.lst`

Model Inference

The accuracy.sh script is shown as below:

#!/bin/sh

source ../../base_config.sh                      # load basic configurations
export SHOW_FPS_LOG=1                            # specify environment variable, print fps level log

${app} \                                         # executable program defined in the accuracy.sh script
  --config_file=workflow_accuracy.json \         # load workflow configuration file of accuracy evaluation
  --log_level=2                                  # specify log level

After the data has been mounted, log in dev board and run the accuracy.sh script in the centernet_resnet101 directory, as shown below:

root@dvb-hynix8G:/userdata/ptq/script/detection/centernet_resnet101# sh accuracy.sh
../../aarch64/bin/example --config_file=workflow_accuracy.json --log_level=2
...
I0419 03:14:51.158655 39555 infer_method.cc:107] Predict DoProcess finished.
I0419 03:14:51.187361 39556 ptq_centernet_post_process_method.cc:558] PTQCenternetPostProcessMethod DoProcess finished, predict result: [{"bbox":[-1.518860,71.691170,574.934631,638.294922],"prob":0.750647,"label":21,"class_name":"
I0118 14:02:43.636204 24782 ptq_centernet_post_process_method.cc:558] PTQCenternetPostProcessMethod DoProcess finished, predict result: [{"bbox":[3.432283,164.936249,157.480042,264.276825],"prob":0.544454,"label":62,"class_name":"
...

Inference results will be saved into the eval.log file dumped by dev board program.

Model Accuracy Computing

Attention

Please perform the accuracy calculation in docker environment or Linux environment.

Accuracy computing is presented in two cases: PTQ model accuracy computing and QAT model accuracy computing.

PTQ Model Accuracy Computing:

For the PTQ model, the model accuracy computing scripts are placed under the ptq/tools/python_tools/accuracy_tools folder, in which:

Script	Description
cls_imagenet_eval.py	The script is used for computing accuracy of classification models evaluated using the ImageNet dataset.
det_coco_eval.py	The script is used for computing the accuracy of models evaluated using the COCO dataset.
seg_cityscapes_eval.py	The script is used for computing the accuracy of segmentation models evaluated using the Cityscapes dataset.
det_voc_eval.py	The script is used for computing the accuracy of detection models using the VOC dataset.

Below we provide you with the description of the different types of PTQ model accuracy computing:

Model Type Description of the Accuracy Computing

Classification Model

Model Type	Description of the Accuracy Computing
Classification Model	Method to compute the accuracy of those models using the ImageNet datasets is shown as below: `python3 cls_imagenet_eval.py --eval-log-path=eval.log \ --gt-file=val.txt` In which: `eval-log-path` refers to inference result file of classification models. `gt-file` refers to the annotation files of ImageNet datasets.
Detection Model	1.Method to compute the accuracy of those models using the COCO dataset is shown as below: `python3 det_coco_eval.py --eval-log-path=eval.log \ --annotation-path=instances_val2017.json` In which: `eval-log-path`refers to inference result file of detection models. `annotation-path`refers to the annotation file of of the COCO dataset. 2.Method to compute the accuracy of those detection models using the VOC dataset is shown as below: `python3 det_voc_eval.py --eval-log-path=eval.log \ --annotation-path=../Annotations \ --val-txt-path=../val.txt` In which: `eval-log-path` refers to the inference result file of detection models. `annotation-path` refers to the annotation file of the VOC dataset. `val-txt-path` refers to the val.txt file in the .../ImageSets/Main folder in the VOC dataset.
Segmentation Model	Method to compute the accuracy of those segmentation models using the Cityscapes dataset is shown as below: `python3 seg_cityscapes_eval.py --eval-log-path=eval.log \ --gt-path=cityscapes/gtFine/val` In which: `eval-log-path` refers to the inference result file of segmentation models. `gt-path` refers to the annotation file of the Cityscapes dataset.

Method to compute the accuracy of those models using the ImageNet datasets is shown as below:

python3 cls_imagenet_eval.py --eval-log-path=eval.log \
                             --gt-file=val.txt

In which:

eval-log-path refers to inference result file of classification models.
gt-file refers to the annotation files of ImageNet datasets.

Detection Model

1.Method to compute the accuracy of those models using the COCO dataset is shown as below:

python3 det_coco_eval.py --eval-log-path=eval.log \
                         --annotation-path=instances_val2017.json

In which:

eval-log-pathrefers to inference result file of detection models.
annotation-pathrefers to the annotation file of of the COCO dataset.

2.Method to compute the accuracy of those detection models using the VOC dataset is shown as below:

python3 det_voc_eval.py --eval-log-path=eval.log \
                        --annotation-path=../Annotations \
                        --val-txt-path=../val.txt

In which:

eval-log-path refers to the inference result file of detection models.
annotation-path refers to the annotation file of the VOC dataset.
val-txt-path refers to the val.txt file in the .../ImageSets/Main folder in the VOC dataset.

Segmentation Model

Method to compute the accuracy of those segmentation models using the Cityscapes dataset is shown as below:

python3 seg_cityscapes_eval.py --eval-log-path=eval.log \
                               --gt-path=cityscapes/gtFine/val

In which:

eval-log-path refers to the inference result file of segmentation models.
gt-path refers to the annotation file of the Cityscapes dataset.

QAT Model Accuracy Computing:

Tip

Location of the accuracy computing scripts: oe package/samples/ai_toolchain/horizon_model_train_sample/scripts. The accuracy computing scripts need to be run in a GPU environment.

Below we provide you with the description of the different types of QAT model accuracy computing:

Dataset Description of the Accuracy Computing

Dataset	Description of the Accuracy Computing
ImageNet Dataset	mixvargenet_imagenet `python3 tools/bpu_validation.py \ -c configs/classification/mixvargenet_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/mixvargenet_imagenet/imagenet_gt.jsonl` mobilenetv1_imagenet `python3 tools/bpu_validation.py \ -c configs/classification/mobilenetv1_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/mobilenetv1_imagenet/imagenet_gt.jsonl` mobilenetv2_imagenet `python3 tools/bpu_validation.py \ -c configs/classification/mobilenetv2_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/mobilenetv2_imagenet/imagenet_gt.jsonl` resnet18_imagenet `python3 tools/bpu_validation.py \ -c configs/classification/resnet18_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/resnet18_imagenet/imagenet_gt.jsonl` resnet50_imagenet `python3 tools/bpu_validation.py \ -c configs/classification/resnet50_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/resnet50_imagenet/imagenet_gt.jsonl` horizon_swin_transformer_imagenet `python3 tools/bpu_validation.py \ -c configs/classification/horizon_swin_transformer_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/horizon_swin_transformer_imagenet/imagenet_gt.jsonl` vit_small_imagenet `python3 tools/bpu_validation.py \ -c configs/classification/vit_small_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/vit_small_imagenet/imagenet_gt.jsonl` vargnetv2_imagenet `python3 tools/bpu_validation.py \ -c configs/classification/vargnetv2_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/vargnetv2_imagenet/imagenet_gt.jsonl` In which: `result-path` refers to inference result file of the models. `gt-files-path` refers to the annotation files of ImageNet dataset.
COCO Dataset	fcos_efficientnetb0_mscoco `# the parameter is_acc_run is true in workflow_accuracy.json during inference python3 tools/bpu_validation.py \ -c configs/detection/fcos/fcos_efficientnetb0_mscoco.py \ --result-path eval.log \ --gt-files-path ./data/coco/fcos_efficientnetb0_mscoco/meta.json # the parameter is_acc_run is false in workflow_accuracy.json during inference python3 tools/bpu_validation.py \ -c configs/detection/fcos/fcos_efficientnetb0_mscoco.py \ --result-path eval.log` detr_resnet50_mscoco `python3 tools/bpu_validation.py \ -c configs/detection/detr/detr_resnet50_mscoco.py \ --result-path eval.log \ --gt-files-path ./data/coco/detr_resnet50_mscoco/meta.json` detr_efficientnetb3_mscoco `python3 tools/bpu_validation.py \ -c configs/detection/detr/detr_efficientnetb3_mscoco.py \ --result-path eval.log \ --gt-files-path ./data/coco/detr_efficientnetb3_mscoco/meta.json` deform_detr_resnet50_mscoco `python3 tools/bpu_validation.py \ -c configs/detection/deform_detr/deform_detr_resnet50_mscoco.py \ --result-path eval.log \ --gt-files-path ./data/coco/deform_detr_resnet50_mscoco/meta.json` In which: `result-path` refers to inference result file of the models. `gt-files-path` refers to the annotation files of COCO dataset.
Cityscapes Dataset	`python3 tools/bpu_validation.py \ -c configs/segmentation/unet_mobilenetv1_cityscapes.py \ --result-path eval.log \ --gt-files-path ./data/cityscapes/unet_mobilenetv1_cityscapes/gts/` In which: `result-path` refers to inference result file of the model. `gt-files-path` refers to the directory for storing annotation files of Cityscapes dataset.
Kitti3D Dataset	`python3 tools/bpu_validation.py \ -c configs/detection/pointpillars/pointpillars_kitti_car.py \ --result-path eval.log \ --gt-files-path ./data/kitti3d/pointpillars_kitti_car/gt/val_gt_infos.pkl` In which: `result-path` refers to inference result file of the model. `gt-files-path` refers to the annotation file of Kitti3D dataset.
Culane Dataset	`python3 tools/bpu_validation.py \ -c configs/lane_pred/ganet/ganet_mixvargenet_culane.py \ --result-path eval.log \ --gt-files-path .data/culane/ganet_mixvargenet_culane/gt.json` In which: `result-path` refers to inference result file of the model. `gt-files-path` refers to the annotation file of Culane dataset.
Nuscenes Dataset	centerpoint_pointpillar_nuscenes `python3 tools/bpu_validation.py \ -c configs/detection/centerpoint/centerpoint_pointpillar_nuscenes.py \ --result-path eval.log \ --gt-files-path ./data/nuscenes/centerpoint_pointpillar_nuscenes/val_gt_infos.pkl \ --extra-args --meta_dir ./tmp_ori_data/nuscenes` flashocc_henet_lss_occ3d_nuscenes `python3 tools/bpu_validation.py \ -c configs/occ/flashocc_henet_lss_occ3d_nuscenes.py \ --result-path eval.log \ --gt-files-path ./data/nuscenes/flashocc_henet_lss_occ3d_nuscenes/val_gt_infos.pkl` bevformer_tiny_resnet50_detection_nuscenes `python3 tools/bpu_validation.py \ -c configs/bev/bevformer_tiny_resnet50_detection_nuscenes.py \ --result-path eval.log \ --gt-files-path ./data/nuscenes/bevformer_tiny_resnet50_detection_nuscenes/val_gt_infos.pkl` bev_lss_efficientnetb0_multitask_nuscenes `python3 tools/bpu_validation.py \ -c configs/bev/bev_lss_efficientnetb0_multitask_nuscenes.py \ --result-path bev_det_eval.log bev_seg_eval.log \ --gt-files-path ./data/nuscenes/bev_lss_efficientnetb0_multitask_nuscenes/val_gt_infos.pkl` petr_efficientnetb3_nuscenes `python3 tools/bpu_validation.py \ -c configs/bev/petr_efficientnetb3_nuscenes.py \ --result-path eval.log \ --gt-files-path ./data/nuscenes/petr_efficientnetb3_nuscenes/val_gt_infos.pkl` detr3d_efficientnetb3_nuscenes `python3 tools/bpu_validation.py \ -c configs/bev/detr3d_efficientnetb3_nuscenes.py \ --result-path eval.log \ --gt-files-path ./data/nuscenes/detr3d_efficientnetb3_nuscenes/val_gt_infos.pkl` bev_gkt_mixvargenet_multitask_nuscenes `python3 tools/bpu_validation.py \ -c configs/bev/bev_gkt_mixvargenet_multitask_nuscenes.py \ --result-path bev_det_eval.log bev_seg_eval.log \ --gt-files-path ./data/nuscenes/bev_gkt_mixvargenet_multitask_nuscenes/val_gt_infos.pkl` bev_ipm_4d_efficientnetb0_multitask_nuscenes `python3 tools/bpu_validation.py \ -c configs/bev/bev_ipm_4d_efficientnetb0_multitask_nuscenes.py \ --result-path bev_seg_eval.log bev_det_eval.log \ --gt-files-path ./data/nuscenes/bev_ipm_4d_efficientnetb0_multitask_nuscenes/val_gt_infos.pkl` bev_ipm_efficientnetb0_multitask_nuscenes `python3 tools/bpu_validation.py \ -c configs/bev/bev_ipm_efficientnetb0_multitask_nuscenes.py \ --result-path bev_det_eval.log bev_seg_eval.log \ --gt-files-path ./data/nuscenes/bev_ipm_efficientnetb0_multitask_nuscenes/val_gt_infos.pkl` fcos3d_efficientnetb0_nuscenes `python3 tools/bpu_validation.py \ -c configs/detection/fcos3d/fcos3d_efficientnetb0_nuscenes.py \ --result-path eval.log \ --gt-files-path ./tmp_data/nuscenes/v1.0-trainval` centerpoint_mixvargnet_multitask_nuscenes `python3 tools/bpu_validation.py \ -c configs/lidar_multi_task/centerpoint_mixvargnet_multitask_nuscenes.py \ --result-path det_eval.log seg_eval.log \ --gt-files-path ./data/nuscenes/centerpoint_mixvargnet_multitask_nuscenes/val_gt_infos.pkl` maptroe_henet_tinym_bevformer_nuscenes `python3 tools/bpu_validation.py \ -c configs/map/maptroe_henet_tinym_bevformer_nuscenes.py \ --result-path eval.log \ --gt-files-path ./data/nuscenes/maptroe_henet_tinym_bevformer_nuscenes/nuscenes_map_anns_val.json` In which: `result-path` refers to inference result files of the models. `gt-files-path` refers to the annotation files of nuscenes dataset.
Mot17 Dataset	`python3 tools/bpu_validation.py \ -c configs/track_pred/motr_efficientnetb3_mot17.py \ --result-path ./eval_log/ \ --gt-files-path ./tmp_data/mot17/test_gt/` In which: `result-path` refers to the directory for storing inference result files of the model. `gt-files-path` refers to the directory for storing annotation files of Mot17 dataset.
Carfusion Dataset	`python3 tools/bpu_validation.py \ -c configs/keypoint/keypoint_efficientnetb0_carfusion.py \ --result-path eval.log \ --gt-files-path ./data/carfusion/keypoint_efficientnetb0_carfusion/gt.json` In which: `result-path` refers to inference result file of the model. `gt-files-path` refers to the annotation file of Carfusion dataset.
Argoverse1 Dataset	`python3 tools/bpu_validation.py \ -c configs/traj_pred/densetnt_vectornet_argoverse1.py \ --result-path eval.log \ --gt-files-path ./data/argoverse1/densetnt_vectornet_argoverse1/meta` In which: `result-path` refers to inference result file of the model. `gt-files-path` refers to the directory for storing annotation files of Argoverse1 dataset.
Argoverse2 Dataset	`python3 tools/bpu_validation.py \ -c configs/traj_pred/qcnet_oe_argoverse2.py \ --result-path eval.log \ --gt-files-path ./data/argoverse2/qcnet_oe_argoverse2/val` In which: `result-path` refers to inference result file of the model. `gt-files-path` refers to the directory for storing annotation files of Argoverse2 dataset.
SceneFlow Dataset	`python3 tools/bpu_validation.py \ -c configs/disparity_pred/stereonet/stereonetplus_mixvargenet_sceneflow.py \ --result-path eval.log \ --gt-files-path ./data/sceneflow/stereonetplus_mixvargenet_sceneflow/gt/val_gt_infos.pkl` In which: `result-path` refers to inference result file of the model. `gt-files-path` refers to the annotation file of SceneFlow dataset.

ImageNet

Dataset

mixvargenet_imagenet
python3 tools/bpu_validation.py \ -c configs/classification/mixvargenet_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/mixvargenet_imagenet/imagenet_gt.jsonl
mobilenetv1_imagenet
python3 tools/bpu_validation.py \ -c configs/classification/mobilenetv1_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/mobilenetv1_imagenet/imagenet_gt.jsonl
mobilenetv2_imagenet
python3 tools/bpu_validation.py \ -c configs/classification/mobilenetv2_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/mobilenetv2_imagenet/imagenet_gt.jsonl
resnet18_imagenet
python3 tools/bpu_validation.py \ -c configs/classification/resnet18_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/resnet18_imagenet/imagenet_gt.jsonl
resnet50_imagenet
python3 tools/bpu_validation.py \ -c configs/classification/resnet50_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/resnet50_imagenet/imagenet_gt.jsonl
horizon_swin_transformer_imagenet
python3 tools/bpu_validation.py \ -c configs/classification/horizon_swin_transformer_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/horizon_swin_transformer_imagenet/imagenet_gt.jsonl
vit_small_imagenet
python3 tools/bpu_validation.py \ -c configs/classification/vit_small_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/vit_small_imagenet/imagenet_gt.jsonl
vargnetv2_imagenet
python3 tools/bpu_validation.py \ -c configs/classification/vargnetv2_imagenet.py \ --result-path eval.log \ --gt-files-path ./data/imagenet/vargnetv2_imagenet/imagenet_gt.jsonl

In which:

result-path refers to inference result file of the models.
gt-files-path refers to the annotation files of ImageNet dataset.

COCO

Dataset

fcos_efficientnetb0_mscoco
# the parameter is_acc_run is true in workflow_accuracy.json during inference python3 tools/bpu_validation.py \ -c configs/detection/fcos/fcos_efficientnetb0_mscoco.py \ --result-path eval.log \ --gt-files-path ./data/coco/fcos_efficientnetb0_mscoco/meta.json # the parameter is_acc_run is false in workflow_accuracy.json during inference python3 tools/bpu_validation.py \ -c configs/detection/fcos/fcos_efficientnetb0_mscoco.py \ --result-path eval.log
detr_resnet50_mscoco
python3 tools/bpu_validation.py \ -c configs/detection/detr/detr_resnet50_mscoco.py \ --result-path eval.log \ --gt-files-path ./data/coco/detr_resnet50_mscoco/meta.json
detr_efficientnetb3_mscoco
python3 tools/bpu_validation.py \ -c configs/detection/detr/detr_efficientnetb3_mscoco.py \ --result-path eval.log \ --gt-files-path ./data/coco/detr_efficientnetb3_mscoco/meta.json
deform_detr_resnet50_mscoco
python3 tools/bpu_validation.py \ -c configs/detection/deform_detr/deform_detr_resnet50_mscoco.py \ --result-path eval.log \ --gt-files-path ./data/coco/deform_detr_resnet50_mscoco/meta.json

In which:

result-path refers to inference result file of the models.
gt-files-path refers to the annotation files of COCO dataset.

Cityscapes

Dataset

python3 tools/bpu_validation.py \
            -c configs/segmentation/unet_mobilenetv1_cityscapes.py \
            --result-path eval.log \
            --gt-files-path ./data/cityscapes/unet_mobilenetv1_cityscapes/gts/

In which:

result-path refers to inference result file of the model.
gt-files-path refers to the directory for storing annotation files of Cityscapes dataset.

Kitti3D

Dataset

python3 tools/bpu_validation.py \
            -c configs/detection/pointpillars/pointpillars_kitti_car.py \
            --result-path eval.log \
            --gt-files-path ./data/kitti3d/pointpillars_kitti_car/gt/val_gt_infos.pkl

In which:

result-path refers to inference result file of the model.
gt-files-path refers to the annotation file of Kitti3D dataset.

Culane

Dataset

python3 tools/bpu_validation.py \
            -c configs/lane_pred/ganet/ganet_mixvargenet_culane.py \
            --result-path eval.log \
            --gt-files-path .data/culane/ganet_mixvargenet_culane/gt.json

In which:

result-path refers to inference result file of the model.
gt-files-path refers to the annotation file of Culane dataset.

Nuscenes

Dataset

centerpoint_pointpillar_nuscenes
python3 tools/bpu_validation.py \ -c configs/detection/centerpoint/centerpoint_pointpillar_nuscenes.py \ --result-path eval.log \ --gt-files-path ./data/nuscenes/centerpoint_pointpillar_nuscenes/val_gt_infos.pkl \ --extra-args --meta_dir ./tmp_ori_data/nuscenes
flashocc_henet_lss_occ3d_nuscenes
python3 tools/bpu_validation.py \ -c configs/occ/flashocc_henet_lss_occ3d_nuscenes.py \ --result-path eval.log \ --gt-files-path ./data/nuscenes/flashocc_henet_lss_occ3d_nuscenes/val_gt_infos.pkl
bevformer_tiny_resnet50_detection_nuscenes
python3 tools/bpu_validation.py \ -c configs/bev/bevformer_tiny_resnet50_detection_nuscenes.py \ --result-path eval.log \ --gt-files-path ./data/nuscenes/bevformer_tiny_resnet50_detection_nuscenes/val_gt_infos.pkl
bev_lss_efficientnetb0_multitask_nuscenes
python3 tools/bpu_validation.py \ -c configs/bev/bev_lss_efficientnetb0_multitask_nuscenes.py \ --result-path bev_det_eval.log bev_seg_eval.log \ --gt-files-path ./data/nuscenes/bev_lss_efficientnetb0_multitask_nuscenes/val_gt_infos.pkl
petr_efficientnetb3_nuscenes
python3 tools/bpu_validation.py \ -c configs/bev/petr_efficientnetb3_nuscenes.py \ --result-path eval.log \ --gt-files-path ./data/nuscenes/petr_efficientnetb3_nuscenes/val_gt_infos.pkl
detr3d_efficientnetb3_nuscenes
python3 tools/bpu_validation.py \ -c configs/bev/detr3d_efficientnetb3_nuscenes.py \ --result-path eval.log \ --gt-files-path ./data/nuscenes/detr3d_efficientnetb3_nuscenes/val_gt_infos.pkl
bev_gkt_mixvargenet_multitask_nuscenes
python3 tools/bpu_validation.py \ -c configs/bev/bev_gkt_mixvargenet_multitask_nuscenes.py \ --result-path bev_det_eval.log bev_seg_eval.log \ --gt-files-path ./data/nuscenes/bev_gkt_mixvargenet_multitask_nuscenes/val_gt_infos.pkl
bev_ipm_4d_efficientnetb0_multitask_nuscenes
python3 tools/bpu_validation.py \ -c configs/bev/bev_ipm_4d_efficientnetb0_multitask_nuscenes.py \ --result-path bev_seg_eval.log bev_det_eval.log \ --gt-files-path ./data/nuscenes/bev_ipm_4d_efficientnetb0_multitask_nuscenes/val_gt_infos.pkl
bev_ipm_efficientnetb0_multitask_nuscenes
python3 tools/bpu_validation.py \ -c configs/bev/bev_ipm_efficientnetb0_multitask_nuscenes.py \ --result-path bev_det_eval.log bev_seg_eval.log \ --gt-files-path ./data/nuscenes/bev_ipm_efficientnetb0_multitask_nuscenes/val_gt_infos.pkl
fcos3d_efficientnetb0_nuscenes
python3 tools/bpu_validation.py \ -c configs/detection/fcos3d/fcos3d_efficientnetb0_nuscenes.py \ --result-path eval.log \ --gt-files-path ./tmp_data/nuscenes/v1.0-trainval
centerpoint_mixvargnet_multitask_nuscenes
python3 tools/bpu_validation.py \ -c configs/lidar_multi_task/centerpoint_mixvargnet_multitask_nuscenes.py \ --result-path det_eval.log seg_eval.log \ --gt-files-path ./data/nuscenes/centerpoint_mixvargnet_multitask_nuscenes/val_gt_infos.pkl
maptroe_henet_tinym_bevformer_nuscenes
python3 tools/bpu_validation.py \ -c configs/map/maptroe_henet_tinym_bevformer_nuscenes.py \ --result-path eval.log \ --gt-files-path ./data/nuscenes/maptroe_henet_tinym_bevformer_nuscenes/nuscenes_map_anns_val.json

In which:

result-path refers to inference result files of the models.
gt-files-path refers to the annotation files of nuscenes dataset.

Mot17

Dataset

python3 tools/bpu_validation.py \
            -c configs/track_pred/motr_efficientnetb3_mot17.py \
            --result-path ./eval_log/ \
            --gt-files-path ./tmp_data/mot17/test_gt/

In which:

result-path refers to the directory for storing inference result files of the model.
gt-files-path refers to the directory for storing annotation files of Mot17 dataset.

Carfusion

Dataset

python3 tools/bpu_validation.py \
            -c configs/keypoint/keypoint_efficientnetb0_carfusion.py \
            --result-path eval.log \
            --gt-files-path ./data/carfusion/keypoint_efficientnetb0_carfusion/gt.json

In which:

result-path refers to inference result file of the model.
gt-files-path refers to the annotation file of Carfusion dataset.

Argoverse1

Dataset

python3 tools/bpu_validation.py \
            -c configs/traj_pred/densetnt_vectornet_argoverse1.py \
            --result-path eval.log \
            --gt-files-path ./data/argoverse1/densetnt_vectornet_argoverse1/meta

In which:

result-path refers to inference result file of the model.
gt-files-path refers to the directory for storing annotation files of Argoverse1 dataset.

Argoverse2

Dataset

python3 tools/bpu_validation.py \
            -c configs/traj_pred/qcnet_oe_argoverse2.py \
            --result-path eval.log \
            --gt-files-path ./data/argoverse2/qcnet_oe_argoverse2/val

In which:

result-path refers to inference result file of the model.
gt-files-path refers to the directory for storing annotation files of Argoverse2 dataset.

SceneFlow

Dataset

python3 tools/bpu_validation.py \
            -c configs/disparity_pred/stereonet/stereonetplus_mixvargenet_sceneflow.py \
            --result-path eval.log \
            --gt-files-path ./data/sceneflow/stereonetplus_mixvargenet_sceneflow/gt/val_gt_infos.pkl

In which:

result-path refers to inference result file of the model.
gt-files-path refers to the annotation file of SceneFlow dataset.

Model Integration

Pre-processing

You can add model pre-processing as needed and deploy it to CPU or DSP, taking centerpoint_pointpillar_nuscenes as an example:

Add the preprocessing file qat_centerpoint_preprocess_method.cc and the header file qat_centerpoint_preprocess_method.h.
Add model preprocessing configuration file.

Add Pre-processing File and Header File

The pre-processing qat_centerpoint_preprocess_method.cc files are placed under the ai_benchmark/code/src/method/ folder. While the header file qat_centerpoint_preprocess_method.h files are placed under the ai_benchmark/code/include/method/ floder.

|── ai_benchmark
|  |── code                                                 # source code of samples
|  |  |── include
|  |  |  |── method                                         # add your header files into this folder
|  |  |  |  |── qat_centerpoint_preprocess_method.h
|  |  |  |  |── ...
|  |  |── src
|  |  |  |── method                                         # add your .cc preprocess files into this folder
|  |  |  |  |── qat_centerpoint_preprocess_method.cc
|  |  |  |  |── ...

Add Preprocessing Configuration File

|── ai_benchmark
|  |── runtime/qat/script                                  # sample script folder
|  |  |── config
|  |  |  |── preprocess
|  |  |  |  |── centerpoint_preprocess_5dim.json           # pre-processing configuration file

Preprocess of centerpoint_pointpillar_nuscenes can deploy to CPU or DSP depends on whether you config the run_on_dsp parameter in the centerpoint_pointpillar_5dim.json. If run_on_dsp in config file is set to true then preprocess will be running on DSP otherwise it running on CPU.

To Evalute the Latency of Preprocessing

Run sh latency.sh to evaluate single frame latency of preprocess, as shown below:

I0807 15:21:15.665417 21457 output_plugin.cc:95]  Pre process latency: [avg:  11.338ms,  max:  14.744ms,  min:  8.200ms], Infer latency:  [avg:  49.327ms,  max:  51.065ms,  min:  48.409ms], Post process latency: [avg:  9.681ms,  max:  9.886ms,  min:  9.544ms].

In which:

Pre process denotes the time consumption of pre-processing.
Infer denotes the time consumption of model inference.
Post process denotes the time consumption of post-processing.

Postprocessing

Post-processing consists of 2 steps. Let's take integration of CenterNet model as an example:

Add the post-processing file ptq_centernet_post_process_method.cc and the header file ptq_centernet_post_process_method.h.
Add a model execution script and a configuration file.

Add Post-processing File

Post-processing code file can reuse any post-processing files in the src/method directory. You only need to modify the InitFromJsonString function and the PostProcess function.

The InitFromJsonString function is used for loading the post-processing related parameters in the workflow.json. You can customize the corresponding input parameters. The PostProcess function is used for implementing post-processing logic.

The post-processing ptq_centernet_post_process_method.cc files are placed under the ai_benchmark/code/src/method/ folder. While the header files ptq_centernet_post_process_method.h are placed under the ai_benchmark/code/include/method/ folder.

|── ai_benchmark
|  |── code                                                 # source code of samples
|  |  |── include
|  |  |  |── method                                         # add your header files into this folder
|  |  |  |  |── ptq_centernet_post_process_method.h
|  |  |  |  |── ...
|  |  |── src
|  |  |  |── method                                         # add your .cc postprocess files into this folder
|  |  |  |  |── ptq_centernet_post_process_method.cc
|  |  |  |  |── ...

Add Model Execution and Configuration Files

Directory structure of scripts is shown as below:

The centerpoint_pointpillar_nuscenes model:

|── ai_benchmark | |── runtime/qat/script # sample script folder | | |── detection | | | |── centerpoint_pointpillar_nuscenes | | | | |── accuracy.sh # accuracy evaluation script | | | | |── fps.sh # performance evaluation script | | | | |── latency.sh # single-frame latency sample script | | | | |── README.md # dsp deployment introduction document | | | | |── workflow_accuracy # accuracy configuration folder | | | | |── workflow_fps.json # performance configuration file | | | | |── workflow_latency.json # single-frame latency configuration file

To process on DSP, you need to execute dsp_deploy.sh to deploy the DSP environment. For a detailed introduction to dsp deployment, please refer to README.md.
The motr_efficientnetb3_mot17 model:

|── ai_benchmark | |── runtime/qat/script # sample script folder | | |── tracking | | | |── motr_efficientnetb3_mot17 | | | | |── accuracy.sh # accuracy evaluation script | | | | |── fps.sh # performance evaluation script | | | | |── generate_acc_lst.sh # generate accuracy lst script | | | | |── latency.sh # single-frame latency sample script | | | | |── workflow_accuracy # accuracy configuration folder | | | | |── workflow_fps.json # performance configuration file | | | | |── workflow_latency.json # single-frame latency configuration file
The models except for the centerpoint_pointpillar_nuscenes and motr_efficientnetb3_mot17:

|── ai_benchmark | |── runtime/ptq/script # sample script folder | | |── detection | | | |── centernet_resnet101 | | | | |── accuracy.sh # accuracy evaluation script | | | | |── fps.sh # performance evaluation script | | | | |── latency.sh # single-frame latency sample script | | | | |── workflow_accuracy.json # accuracy configuration file | | | | |── workflow_fps.json # performance configuration file | | | | |── workflow_latency.json # single-frame latency configuration file

Helper Tools

Log

There are 2 types of logs: sample Log and DNN Log. Wherein, sample log refers to the log in the AI Benchmark Sample Package deliverables, while DNN log refers to the log in the embedded runtime library. You can specify logs as needed.

Sample Log

Log level.

Both glog and vlog are used in sample log and there are 4 customized log levels:

0: SYSTEM level, this log level is used for generating error information in sample code.
1: REPORT level, this log level is used for generating performance data in sample code.
2: DETAIL level, this log level is used for generating current system status in sample code.
3: DEBUG level, this log level is used for generating debugging information in sample code.

Set log levels.

Rules to set log levels: The default ranks of log level: DEBUG>DETAIL>REPORT>SYSTEM, the higher the level, the more logs will be output. That is, if you set a high level, the logs corresponding to your own level and the level below it will be output.

When running samples, specify the log_level parameter to set log levels. For example, if log_level=0, then SYSTEM log should be dumped; else if log_level=3, then DEBUG, DETAIL, REPORT and SYSTEM logs should be dumped.

dnn Log

For the configuration of dnn logs, please read the Configuration Info section in the Model Inference API Instruction.

OP Time Consumption

Overview

Use the HB_DNN_PROFILER_LOG_PATH environment variable to specify statistics of OP performance. Types and values of this environment variable are described as below:

HB_DNN_PROFILER_LOG_PATH=${path}: denotes the output path of OP node. After the program is executed, a profiler.log file should be generated.

Sample

Taking the mobilenetv1 as an example, as shown in the following code block: Start 1 threads to run the model at the same time, set export HB_DNN_PROFILER_LOG_PATH=. /, then the profiler.log file will output the performance data of the OPs.

The output information contains model_latency and task_latency. Wherein, model_latency contains the time consumption required to run each operator of the model; while task_latency contains the time consumption of each task of the model.

Dump Tool

Enable the HB_DNN_DUMP_PATH environment variable to dump the input and output of each node in inference process. The dump tool can check if there are consistency problems between simulator and real machine, i.e. Whether the output of the real machine and the simulator are exactly the same, given the same model and the same inputs.

The hb_compile Tool

QAT

qconfig_setter

Model Export

Horizon Operator

Model Inference API Introduction

Data Type and Data Structure

Function Interface

Model Inference Tool Introduction

The hrt_model_exec Tool Introduction

The hbm_infer Tool Introduction

Data Structure

Functional Interface

Operator Support List

Operator BPU Constraint List

#AI Benchmark User Guide

#Release Description

#Sample Code Package

#Sample Models

#Public Datasets

#Environment Configuration

#How to Use

#Evaluation Scripts

#Parameters of json Configuration File

#input_config

#output_config

#Workflow Configuration

#Performance Evaluation

#How to Use Performance Evaluation Scripts

#About Command-line Parameters

#Result Visualization

#Model Accuracy Evaluation

#Data Pre-processing

#Model Mounting

#Generate lst Files

#Model Inference

#Model Accuracy Computing

#Model Integration

#Pre-processing

#Add Pre-processing File and Header File

#Add Preprocessing Configuration File

#To Evalute the Latency of Preprocessing

#Postprocessing

#Add Post-processing File

#Add Model Execution and Configuration Files

#Helper Tools

#Log

#Sample Log

#dnn Log

#OP Time Consumption

#Overview

#Sample

#Dump Tool

AI Benchmark User Guide

Release Description

Sample Code Package

Sample Models

Public Datasets

Environment Configuration

How to Use

Evaluation Scripts

Parameters of json Configuration File

input_config

output_config

Workflow Configuration

Performance Evaluation

How to Use Performance Evaluation Scripts

About Command-line Parameters

Result Visualization

Model Accuracy Evaluation

Data Pre-processing

Model Mounting

Generate lst Files

Model Inference

Model Accuracy Computing

Model Integration

Pre-processing

Add Pre-processing File and Header File

Add Preprocessing Configuration File

To Evalute the Latency of Preprocessing

Postprocessing

Add Post-processing File

Add Model Execution and Configuration Files

Helper Tools

Log

Sample Log

dnn Log

OP Time Consumption

Overview

Sample

Dump Tool