RKNN-Toolkit2 Model Conversion API Reference

1.API Details

1.1 RKNN Initialization and Release

This section describes 1.1 RKNN Initialization and Release. Related API or structure: firstRKNN(), release().

This section describes 1.1 RKNN Initialization and Release.

Example:

# Output detailed logs
from rknn.api import RKNN
rknn = RKNN(verbose=True)
# ...
rknn.release()

1.2 Model Configuration

This section describes 1.2 Model Configuration.

Item	Content
API	config
Description	modelconversionParameters set。
Parameters	Description of Parameters.
Return Value	None.

config Parameter Details

Parameters	Description
mean_values	Description of mean_values. Related values: None.
std_values	Description of std_values. Related values: None.
quant_img_RGB2BGR	Description of quant_img_RGB2BGR. Related values: True True False True.
quantized_dtype	Description of quantized_dtype. Related values: w8a8 w4a16 w8a16 w4a8 w16a16 w16a16 w8a8 w8a8 w4a16 w8a16 w4a8 w16a16 w16a16.
quantized_algorithm	Description of quantized_algorithm. Related values: w4a16 w8a16 w4a16 w8a16.
quantized_method	Description of quantized_method. Related values: w4a16.
float_dtype	Description of float_dtype. Related values: float16 float16.
optimization_level	Description of optimization_level.
target_platform	Description of target_platform. Related values: None.
custom_string	Description of custom_string. Related values: None.
remove_weight	Description of remove_weight. Related values: False.
compress_weight	Description of compress_weight. Related values: False.
single_core_mode	Description of single_core_mode. Related values: False.
model_pruning	Description of model_pruning. Related values: False.
op_target	Description of op_target. Related values: None.
dynamic_input	Description of dynamic_input. Related values: None.
quantize_weight	Description of quantize_weight. Related values: False False.
remove_reshape	Description of remove_reshape. Related values: False.
sparse_infer	Description of sparse_infer. Related values: False.
enable_flash_attention	Description of enable_flash_attention. Related values: False.
auto_hybrid_cos_thresh	Description of auto_hybrid_cos_thresh.
auto_hybrid_euc_thresh	Description of auto_hybrid_euc_thresh. Related values: None.

Example:

# model config
rknn.config(mean_values=[[103.94, 116.78, 123.68]],
            std_values=[[58.82, 58.82, 58.82]],
            quant_img_RGB2BGR=True,
            target_platform='rk3566')

1.3 Model Loading

This section describes 1.3 Model Loading.

1.3.1 Caffe Model Loading Interface

Item	Content
API	load_caffe
Description	Description of Description.
Parameters	Description of Parameters. Related values: None.
Return Value	0：importsuccess。 -1：importfailure。

Example:

# Load the mobilenet_v2 model from the current path
ret = rknn.load_caffe(model='./mobilenet_v2.prototxt',
                      blobs='./mobilenet_v2.caffemodel')

1.3.2 TensorFlow Model Loading Interface

Item	Content
API	load_tensorflow
Description	Description of Description.
Parameters	Description of Parameters. Related values: NCHW False NHWC.
Return Value	0：importsuccess。 -1：importfailure。

Example:

# Load the ssd_mobilenet_v1_coco_2017_11_17 model from the current directory
ret = rknn.load_tensorflow(tf_pb='./ssd_mobilenet_v1_coco_2017_11_17.pb',
                           inputs=['Preprocessor/sub'],
                           outputs=['concat', 'concat_1'],
                           input_size_list=[[300, 300, 3]])

1.3.3 TensorFlow Lite Model Loading Interface

Item	Content
API	load_tflite
Description	Description of Description.
Parameters	Description of Parameters. Related values: NCHW False NHWC.
Return Value	0：importsuccess。 -1：importfailure。

Example:

# Load the mobilenet_v1 model from the current directory
ret = rknn.load_tflite(model='./mobilenet_v1.tflite')

1.3.4 ONNX Model Loading

Item	Content
API	load_onnx
Description	ONNXmodel load。
Parameters	Description of Parameters. Related values: None None None None None array( None.
Return Value	0：importsuccess。 -1：importfailure。

Example:

# Load the arcface model from the current directory
ret = rknn.load_onnx(model='./arcface.onnx')

1.3.5 DarkNet Model Loading Interface

Item	Content
API	load_darknet
Description	Description of Description.
Parameters	Description of Parameters.
Return Value	0：importsuccess。 -1：importfailure。

Example:

# Load the yolov3-tiny model from the current directory
ret = rknn.load_darknet(model='./yolov3-tiny.cfg',
                        weight='./yolov3.weights')

1.3.6 PyTorch Model Loading Interface

Item	Content
API	load_pytorch
Description	Description of Description.
Parameters	Description of Parameters.
Return Value	0：importsuccess。 -1：importfailure。

Example:

# Load the resnet18 model from the current directory
ret = rknn.load_pytorch(model='./resnet18.pt',
                        input_size_list=[[1,3,224,224]])

1.4 Build RKNN Model

Item	Content
API	build
Description	RKNNmodel build。
Parameters	Description of Parameters. Related values: True None False w8a8.
Return Value	0：buildsuccess。 -1：buildfailure。

Example:

# Build the RKNN model and perform quantization
ret = rknn.build(do_quantization=True, dataset='./dataset.txt')

1.5 Export RKNN Model

This section describes 1.5 Export RKNN Model.

Item	Content
API	export_rknn
Description	RKNNmodel specifyfile（.rknnextension） in save。
Parameters	Description of Parameters.
Return Value	0：exportsuccess。 -1：exportfailure。

Example:

# Save the built RKNN model as mobilenet_v1.rknn in the current path
ret = rknn.export_rknn(export_path='./mobilenet_v1.rknn')

1.6 Load RKNN Model

Item	Content
API	load_rknn
Description	Description of Description.
Parameters	path：RKNNmodelfile of path。
Return Value	0：loadsuccess。 -1：loadfailure。

Example:

# Load the mobilenet_v1.rknn model from the current path
ret = rknn.load_rknn(path='./mobilenet_v1.rknn')

1.7 Initialize Runtime Environment

This section describes 1.7 Initialize Runtime Environment.

Item	Content
API	init_runtime
Description	Description of Description.
Parameters	Description of Parameters. Related values: None None None False False False.
Return Value	Description of Return Value.

Example:

# Initialize the runtime environment
ret = rknn.init_runtime(target='rk3566')

1.8 Model Inference

This section describes 1.8 Model Inference.

Item	Content
API	inference
Description	Description of Description.
Parameters	Description of Parameters. Related values: None NHWC None.
Return Value	Description of Return Value.

This section describes 1.8 Model Inference.

# Run image inference with the model and obtain the top-5 results
outputs = rknn.inference(inputs=[img])
show_outputs(outputs)

This section describes 1.8 Model Inference.

-----TOP 5-----
[ 156] score:0.928223 class:"Shih-Tzu"
[ 155] score:0.063171 class:"Pekinese, Pekingese, Peke"
[ 205] score:0.004299 class:"Lhasa, Lhasa apso"
[ 284] score:0.003096 class:"Persian cat"
[ 285] score:0.000171 class:"Siamese cat, Siamese

1.9 Model Performance Evaluation

Item	Content
API	eval_perf
Description	Description of Description. Related values: False True.
Parameters	Description of Parameters. Related values: True True.
Return Value	perf_result：performanceinformation（string）。

Example:

# Evaluate model performance
perf_detail = rknn.eval_perf()

1.10 Get Memory Usage

Item	Content
API	eval_memory
Description	Description of Description.
Parameters	Description of Parameters. Related values: True.
Return Value	Description of Return Value.

Example:

# Evaluate model memory usage
memory_detail = rknn.eval_memory()

This section describes 1.10 Get Memory Usage.

======================================================
Memory Profile Info Dump
======================================================
NPU model memory detail(bytes):
Weight Memory: 3.53 MiB
Internal Tensor Memory: 1.67 MiB
Other Memory: 473.00 KiB
Total Memory: 5.66 MiB

INFO: When evaluating memory usage, we need consider
      the size of model, current model size is: 4.09 MiB
======================================================

1.11 Query SDK Version

Item	Content
API	get_sdk_version
Description	Description of Description.
Parameters	None.
Return Value	sdk_version：APIanddriver of versioninformation。type string。

Example:

# Get SDK version information
sdk_version = rknn.get_sdk_version()
print(sdk_version)

This section describes 1.11 Query SDK Version.

==============================================
RKNN VERSION:
API: 1.5.2 (8babfea build@2023-08-25T02:31:12)
DRV: rknn_server: 1.5.2 (8babfea build@2023-08-25T10:30:12)
DRV: rknnrt: 1.5.3b13 (42cbca6f5@2023-10-27T10:13:21)
==============================================

1.12 Hybrid Quantization

1.12.1 hybrid_quantization_step1

This section describes 1.12.1 hybrid_quantization_step1.

Item	Content
API	hybrid_quantization_step1
Description	Description of Description.
Parameters	Description of Parameters. Related values: False None.
Return Value	0：success。 -1：failure。

Example:

# Call hybrid_quantization_step1 to generate the quantization configuration file
ret = rknn.hybrid_quantization_step1(dataset='./dataset.txt')

1.12.2 hybrid_quantization_step2

This section describes 1.12.2 hybrid_quantization_step2.

Item	Content
API	hybrid_quantization_step2
Description	Description of Description.
Parameters	Description of Parameters.
Return Value	0：success。 -1：failure。

Example:

# Call hybrid_quantization_step2 to generate hybrid quantized RKNN model
ret = rknn.hybrid_quantization_step2(
    model_input='./ssd_mobilenet_v2.model',
    data_input='./ssd_mobilenet_v2.data',
    model_quantization_cfg='./ssd_mobilenet_v2.quantization.cfg')

1.13 Quantization Accuracy Analysis

This section describes 1.13 Quantization Accuracy Analysis.

Item	Content
API	accuracy_analysis
Description	Description of Description.
Parameters	Description of Parameters. Related values: None None.
Return Value	0：success。 -1：failure。

Example:

# Accuracy analysis
ret = rknn.accuracy_analysis(inputs=['./dog_224x224.jpg'])

1.14 Get Device List

Item	Content
API	list_devices
Description	Description of Description.
Parameters	None.
Return Value	Description of Return Value.

Example:

rknn.list_devices()

This section describes 1.14 Get Device List.

*************************
all device(s) with adb mode:
VD46C3KM6N
*************************

This section describes 1.14 Get Device List.

1.15 Export Encrypted Model

This section describes 1.15 Export Encrypted Model.

Item	Content
API	export_encrypted_rknn_model
Description	Description of Description.
Parameters	Description of Parameters. Related values: None.
Return Value	0：success。 -1：failure。

Example:

ret = rknn.export_encrypted_rknn_model('test.rknn')

1.16 Register Custom Operator

This section describes 1.16 Register Custom Operator.

Item	Content
API	reg_custom_op
Description	Description of Description.
Parameters	Description of Parameters.
Return Value	0：success。 -1：failure。

Example:

import numpy as np
from rknn.api.custom_op import get_node_attr

class cstSoftmax:
    op_type = 'cstSoftmax'

    def shape_infer(self, node, in_shapes, in_dtypes):
        out_shapes = in_shapes.copy()
        out_dtypes = in_dtypes.copy()
        return out_shapes, out_dtypes

    def compute(self, node, inputs):
        x = inputs[0]
        axis = get_node_attr(node, 'axis')
        x_max = np.max(x, axis=axis, keepdims=True)
        tmp = np.exp(x - x_max)
        s = np.sum(tmp, axis=axis, keepdims=True)
        outputs = [tmp / s]
        return outputs

ret = rknn.reg_custom_op(cstSoftmax)

1.17 Generate C++ Deployment Example

Item	Content
API	codegen
Description	Description of Description.
Parameters	Description of Parameters. Related values: True False.
Return Value	0：success。 -1：failure。

Example:

ret = rknn.codegen(output_path='./rknn_app_demo',
                   inputs=['./mobilenet_v2/dog_224x224.jpg'],
                   overwrite=True)