模型推理
c接口详细介绍请阅读NNToolChain.pdf 和 BMLib_User_Guide.pdf 。
python接口详细介绍请阅读Sophon_Inference_zh.pdf
BMRuntime用于读取BMCompiler的编译输出(.bmodel)，驱动其在BITMAIN TPU芯片中执行。BMRuntime向用户提供了丰富的接口，便于用户移植算法，其软件架构如下:
Bmruntime实现了c/c++/python，本章主要介绍c和python常用接口，本章主要介绍如下内容：
BMLIB 接口：用于设备管理
BMruntime的c语言接口
BMLIB和BMruntime的python接口介绍
1. BMLib模块c接口介绍
BMLIB接口
用于设备管理，不属于BMRuntime，但需要配合使用，所以先介绍
BMLIB接口是C语言接口，对应的头文件是bmlib_runtime.h，对应的lib库为libbmlib.so,
BMLIB接口用于设备管理，包括设备内存的管理。
BMLIB的接口很多，这里介绍应用程序通常需要用到的接口。
bm_dev_request
用于请求一个设备，得到设备句柄handle。其他设备接口，都需要指定这个设备句柄。
其中devid表示设备号，在PCIE模式下，存在多个设备时可以用于选择对应的设备；在SoC模式下，请指定为0。
/** * @name bm_dev_request * @brief To create a handle for the given device * @ingroup bmlib_runtime * * @param [out] handle The created handle * @param [in] devid Specify on which device to create handle * @retval BM_SUCCESS Succeeds. * Other code Fails. */ bm_status_t bm_dev_request(bm_handle_t *handle, int devid);bm_dev_free
用于释放一个设备。通常应用程序开始需要请求一个设备，退出前释放这个设备。
/** * @name bm_dev_free * @brief To free a handle * @param [in] handle The handle to free */ void bm_dev_free(bm_handle_t handle);
2. BMRuntime模块c接口介绍
对应的头文件为bmruntime_interface.h，对应的lib库为libbmrt.so。
用户程序使用C接口时建议使用该接口，该接口支持多种shape的静态编译网络，支持动态编译网络。
bmrt_create
/** * @name bmrt_create * @brief To create the bmruntime with bm_handle. * This API creates the bmruntime. It returns a void* pointer which is the pointer * of bmruntime. Device id is set when get bm_handle; * @param [in] bm_handle bm handle. It must be initialized by using bmlib. * @retval void* the pointer of bmruntime */ void* bmrt_create(bm_handle_t bm_handle);bmrt_destroy
/** * @name bmrt_destroy * @brief To destroy the bmruntime pointer * @ingroup bmruntime * This API destroy the bmruntime. * @param [in] p_bmrt Bmruntime that had been created */ void bmrt_destroy(void* p_bmrt);bmrt_load_bmodel
加载bmodel文件，加载后bmruntime中就会存在若干网络的数据，后续可以对网络进行推理。
/** * @name bmrt_load_bmodel * @brief To load the bmodel which is created by BM compiler * This API is to load bmodel created by BM compiler. * After loading bmodel, we can run the inference of neuron network. * @param [in] p_bmrt Bmruntime that had been created * @param [in] bmodel_path Bmodel file directory. * @retval true Load context sucess. * @retval false Load context failed. */ bool bmrt_load_bmodel(void* p_bmrt, const char *bmodel_path);bmrt_load_bmodel_data
加载bmodel，不同于bmrt_load_bmodel，它的bmodel数据存在内存中
/* Parameters: [in] p_bmrt - Bmruntime that had been created. [in] bmodel_data - Bmodel data pointer to buffer. [in] size - Bmodel data size. Returns: bool - true: success; false: failed. */ bool bmrt_load_bmodel_data(void* p_bmrt, const void * bmodel_data, size_t size);bmrt_get_network_info
bmrt_get_network_info根据网络名，得到某个网络的信息
/* bm_stage_info_t holds input shapes and output shapes; every network can contain one or more stages */ typedef struct { bm_shape_t* input_shapes; /* input_shapes[0] / [1] / ... / [input_num-1] */ bm_shape_t* output_shapes; /* output_shapes[0] / [1] / ... / [output_num-1] */ } bm_stage_info_t; /* bm_tensor_info_t holds all information of one net */ typedef struct { const char* name; /* net name */ bool is_dynamic; /* dynamic or static */ int input_num; /* number of inputs */ char const** input_names; /* input_names[0] / [1] / .../ [input_num-1] */ bm_data_type_t* input_dtypes; /* input_dtypes[0] / [1] / .../ [input_num-1] */ float* input_scales; /* input_scales[0] / [1] / .../ [input_num-1] */ int output_num; /* number of outputs */ char const** output_names; /* output_names[0] / [1] / .../ [output_num-1] */ bm_data_type_t* output_dtypes; /* output_dtypes[0] / [1] / .../ [output_num-1] */ float* output_scales; /* output_scales[0] / [1] / .../ [output_num-1] */ int stage_num; /* number of stages */ bm_stage_info_t* stages; /* stages[0] / [1] / ... / [stage_num-1] */ } bm_net_info_t;bm_net_info_t表示一个网络的全部信息，bm_stage_info_t表示该网络支持的不同的shape情况。
/** * @name bmrt_get_network_info * @brief To get network info by net name * @param [in] p_bmrt Bmruntime that had been created * @param [in] net_name Network name * @retval bm_net_info_t* Pointer to net info, needn't free by user; if net name not found, will return NULL. */ const bm_net_info_t* bmrt_get_network_info(void* p_bmrt, const char* net_name);
示例代码：
const char *model_name = "VGG_VOC0712_SSD_300X300_deploy"
const char **net_names = NULL;
bm_handle_t bm_handle;
bm_dev_request(&bm_handle, 0);
void * p_bmrt = bmrt_create(bm_handle);
bool ret = bmrt_load_bmodel(p_bmrt, bmodel.c_str());
std::string bmodel; //bmodel file
int net_num = bmrt_get_network_number(p_bmrt, model_name);
bmrt_get_network_names(p_bmrt, &net_names);
for (int i=0; i<net_num; i++) {
//do somthing here
......
}
free(net_names);
bmrt_destroy(p_bmrt);
bm_dev_free(bm_handle);bmrt_shape_count
接口声明如下：
/* number of shape elements, shape should not be NULL and num_dims should not large than BM_MAX_DIMS_NUM */ uint64_t bmrt_shape_count(const bm_shape_t* shape);可以得到shape的元素个数。
比如num_dims为4，则得到的个数为dims[0]*dims[1]*dims[2]*dims[3]
bm_shape_t 结构介绍：
typedef struct { int num_dims; int dims[BM_MAX_DIMS_NUM]; } bm_shape_t;bm_shape_t表示tensor的shape，目前最大支持8维的tensor。其中num_dims为tensor的实际维度数，dims为各维度值，dims的各维度值从[0]开始，比如(n, c, h, w)四维分别对应(dims[0], dims[1], dims[2], dims[3])。
如果是常量shape，初始化参考如下：
bm_shape_t shape = {4, {4,3,228,228}}; bm_shape_t shape_array[2] = { {4, {4,3,28,28}}, // [0] {2, {2,4}}, // [1] }bm_image_from_mat
if use this function you need to open USE_OPENCV macro in include/bmruntime/bm_wrapper.hpp /** * @name bm_image_from_mat * @brief Convert opencv Mat object to BMCV bm_image object * @param [in] in OPENCV mat object * @param [out] out BMCV bm_image object * @retval true Launch success. * @retval false Launch failed. */ static inline bool bm_image_from_mat (cv::Mat &in, bm_image &out)* @brief Convert opencv multi Mat object to multi BMCV bm_image object static inline bool bm_image_from_mat (std::vector<cv::Mat> &in, std::vector<bm_image> &out)实例代码请参考bmmnsdk2开发包中examples/SSD_object/cpp_cv_bmcv_bmrt
bm_image_from_frame
/** * @name bm_image_from_frame * @brief Convert ffmpeg a avframe object to a BMCV bm_image object * @ingroup bmruntime * * @param [in] bm_handle the low level device handle * @param [in] in a read-only avframe * @param [out] out an uninitialized BMCV bm_image object use bm_image_destroy function to free out parameter until you no longer useing it. * @retval true change success. * @retval false change failed. */ static inline bool bm_image_from_frame (bm_handle_t &bm_handle, AVFrame &in, bm_image &out)/** * @name bm_image_from_frame * @brief Convert ffmpeg avframe to BMCV bm_image object * @ingroup bmruntime * * @param [in] bm_handle the low level device handle * @param [in] in a read-only ffmpeg avframe vector * @param [out] out an uninitialized BMCV bm_image vector use bm_image_destroy function to free out parameter until you no longer useing it. * @retval true change success. * @retval false chaneg failed. */ static inline bool bm_image_from_frame (bm_handle_t &bm_handle, std::vector<AVFrame> &in, std::vector<bm_image> &out)实例代码请参考bmmnsdk2开发包中examples/SSD_object/cpp_ffmpeg_bmcv_bmrt/main.cpp
bm_inference
if use this function you need to open USE_OPENCV macro in include/bmruntime/bm_wrapper.hpp /** * @name bm_inference * @brief A block inference wrapper call * @ingroup bmruntime * * This API supports the neuron nework that is static-compiled or dynamic-compiled * After calling this API, inference on TPU is launched. And the CPU * program will be blocked. * This API support single input && single output, and multi thread safety * * @param [in] p_bmrt Bmruntime that had been created * @param [in] input bm_image of single-input data * @param [in] output Pointer of single-output buffer * @param [in] net_name The name of the neuron network * @param [in] input_shape single-input shape * * @retval true Launch success. * @retval false Launch failed. */ static inline bool bm_inference (void *p_bmrt, bm_image *input, void *output, bm_shape_t input_shape, const char *net_name)* This API support single input && multi output, and multi thread safety static inline bool bm_inference (void *p_bmrt, bm_image *input, std::vector<void*> outputs, bm_shape_t input_shape, const char *net_name)* This API support multiple inputs && multiple outputs, and multi thread safety static inline bool bm_inference (void *p_bmrt, std::vector<bm_image*> inputs, std::vector<void*> outputs, std::vector<bm_shape_t> input_shapes, const char *net_name)实例代码请参考bmmnsdk2开发包中examples/SSD_object/cpp_cv_bmcv_bmrt/main.cpp
3. Python接口
本章节只介绍了用例py_ffmpeg_bmcv_sail中用的的接口函数
更多接口定义请查阅Sophon_Inference_zh.pdf
Engine
def __init__(tpu_id): """ Constructor does not load bmodel. Parameters --------- tpu_id : int TPU ID. You can use bm-smi to see available IDs """load
def load(bmodel_path): Load bmodel from file. Parameters --------- bmodel_path : str Path to bmode """set_io_mode
def set_io_mode(mode): """ Set IOMode for a graph. Parameters --------- mode : sail.IOMode Specified io mode """get_graph_names
def get_graph_names(): """ Get all graph names in the loaded bmodels. Returns ------ graph_names : list Graph names list in loaded context """get_input_names
def get_input_names(graph_name): """ Get all input tensor names of the specified graph. Parameters --------- graph_name : str Specified graph name Returns ------ input_names : list All the input tensor names of the graph """get_output_names
def get_output_names(graph_name): """ Get all output tensor names of the specified graph. Parameters --------- graph_name : str Specified graph name Returns ------ input_names : list All the output tensor names of the graph """sail.IOMode
# Input tensors are in system memory while output tensors are in device memory sail.IOMode.SYSI # Input tensors are in device memory while output tensors are in system memory. sail.IOMode.SYSO # Both input and output tensors are in system memory. sail.IOMode.SYSIO # Both input and output tensors are in device memory. ail.IOMode.DEVIOset_io_mode
def set_io_mode(mode): """ Set IOMode for a graph. Parameters --------- mode : sail.IOMode Specified io mode """sail.Tensor
def __init__(handle, shape, dtype, own_sys_data, own_dev_data): """ Constructor allocates system memory and device memory of the tensor. Parameters --------- handle : sail.Handle Handle instance shape : tuple Tensor shape dytpe : sail.Dtype Data type own_sys_data : bool Indicator of whether own system memory own_dev_data : bool Indicator of whether own device memory """get_input_dtype
def get_input_dtype(graph_name, tensor_name): """ Get scale of an input tensor. Only used for int8 models. Parameters --------- graph_name : str The specified graph name tensor_name : str The specified output tensor name Returns ------ scale: sail.Dtype Data type of the input tensor """get_output_dtype
def get_output_dtype(graph_name, tensor_name): """ Get the shape of an output tensor in a graph. Parameters --------- graph_name : str The specified graph name tensor_name : str The specified output tensor name Returns ------ tensor_shape : list The shape of the tensor """process
def process(graph_name, input_tensors, output_tensors): """ Inference with provided input and output tensors. Parameters --------- graph_name : str The specified graph name input_tensors : dict {str : sail.Tensor} Input tensors managed by user output_tensors : dict {str : sail.Tensor} Output tensors managed by user """get_input_scale
def get_input_scale(graph_name, tensor_name): """ Get scale of an input tensor. Only used for int8 models. Parameters --------- graph_name : str The specified graph name tensor_name : str The specified output tensor name Returns ------ scale: float32 Scale of the input tensor """
Last updated
Was this helpful?