VOXL Camera Server

Overview

VOXL Camera Server publishes MIPI data to MPA pipes for clients to read.

Example services that are clients for voxl-camera-server include voxl-qvio-server, voxl-portal, ROS, or rtsp streams.

HAL3

VOXL Camera Server uses Google’s HAL3 API to access raw camera data from qualcomm’s mm-qcamera-daemon pipeline.

Configuration

Whenever cameras are plugged/unplugged (which should only happen when the board is powered off), make sure to run voxl-configure-cameras with the appropriate argument (or run without an argument to see the help menu). This script will generate a file at /etc/modalai/voxl-camera-server.conf containing the default settings for that camera configuration. If you want to change some aspects of the camera behavior such as disabling/framerate/resolution(for hires)/exposure setpoint you can do so by modifying that file and restarting camera server with systemctl restart voxl-camera-server.

Parameter definitions

name: Name of camera being used. E.G. tracking camera has a name of “tracking”
flip: If the camera is flipped so the frame is rotated
enabled: Is this camera to be enabled by the service - if set to true, then it will be used or streamed
frame_rate: framerate associated to the camera
type: the actual hardware sensor name - in the case of the tracking camera used (provided by ModalAI), it is the ov7251
camera_id: This is the the id of the camera
ae_desired_msv: the desired mean sample value, a.k.a. the average value of pixels that the auto exposure algorithm should try to achieve in frame
ae_mode: Type of autoexposure mode chosen (such as modal algo, isp, lme_hist, lme_msv)
ae_k_p_ns: the desired p_ns for the exposure algorithm
ae_k_i_ns: the desired k_i for the exposure algorithm
ae_max_i: the desired max_i for the exposure algorithm
ae_filter_alpha: a low-pass filter constant that filters the calculated MSV to slow down responses - the filter used is an IIR filter
ae_ignore_fraction: maximum percentage of saturated (255) pixels that will be used in calculation of MSV. If there are more saturated_pixels / total_pixels, then additional saturated pixels are not used to calculate MSV. This helps prevent image getting too dark if there are large blobs of very bright light
ae_slope: ratio that specifies how much gain vs exposure should be changed when trying to achieve desired MSV. Both gain and exposure linearly affect the pixel brightness, but gain and exposure have different effects on the image quality - mostly in the sense that gain affects granularity and that exposure affects motion blur.
ae_exposure_period: controls the duration where the cells of the camera sensor are exposed to light
ae_gain_period: the gain period associated to the auto exposure (think of this as an amplification factor of the pixels)
preview_width: the preview width of the frame being sent to VOXL-PORTAL from the camera
preview_height: the preview height of the frame being sent to VOXL-PORTAL from the camera
snapshot_width: the snapshot width of the frame being taken as a photo/snapshot from the camera
snapshot_height: the snapshot height of the frame being taken as a photo/snapshot from the camera
stream_width: the stream width of the frame streamed to voxl-streamer
stream_height: the stream height of the frame streamed to voxl-streamer
stream_bitrate: the stream bitrate of the frame streamed to voxl-streamer
record_width: the record width of the frame being saved onboard in a .avi file
record_height: the record height of the frame being saved onboard in a .avi file
record_bitrate: the record bitrate of the frame being saved onboard in a .avi file
independent_exposure: the independent exposure value for a stereo pair

Default parameters

Whenever you run voxl-configure-cameras a new default camera server config file is created for that particular camera setup. Here is an example default config file for camera config #6 used on Starling V2.

{
    "version":  0.1,
    "cameras":  [{
            "type": "pmd-tof",
            "name": "tof",
            "enabled":  true,
            "camera_id":    0,
            "fps":  5,
            "en_preview":   true,
            "preview_width":    224,
            "preview_height":   1557,
            "pre_format":   "tof",
            "ae_mode":  "off",
            "standby_enabled":  false,
            "decimator":    5
        }, {
            "type": "imx214",
            "name": "hires",
            "enabled":  true,
            "camera_id":    1,
            "fps":  30,
            "en_preview":   false,
            "preview_width":    640,
            "preview_height":   480,
            "pre_format":   "nv21",
            "en_small_video":   true,
            "small_video_width":    1024,
            "small_video_height":   768,
            "small_venc_mode":  "h265",
            "small_venc_br_ctrl":   "cqp",
            "small_venc_Qfixed":    30,
            "small_venc_Qmin":  15,
            "small_venc_Qmax":  40,
            "small_venc_nPframes":  9,
            "small_venc_mbps":  2,
            "en_large_video":   true,
            "large_video_width":    4096,
            "large_video_height":   2160,
            "large_venc_mode":  "h265",
            "large_venc_br_ctrl":   "cqp",
            "large_venc_Qfixed":    38,
            "large_venc_Qmin":  15,
            "large_venc_Qmax":  50,
            "large_venc_nPframes":  29,
            "large_venc_mbps":  30,
            "en_snapshot":  true,
            "en_snapshot_width":    4160,
            "en_snapshot_height":   3120,
            "ae_mode":  "isp"
        }, {
            "type": "ov7251",
            "name": "tracking",
            "enabled":  true,
            "camera_id":    2,
            "fps":  30,
            "en_preview":   true,
            "preview_width":    640,
            "preview_height":   480,
            "pre_format":   "raw8",
            "ae_mode":  "lme_msv",
            "ae_desired_msv":   60,
            "ae_filter_alpha":  0.600000023841858,
            "ae_ignore_fraction":   0.20000000298023224,
            "ae_slope": 0.05000000074505806,
            "ae_exposure_period":   1,
            "ae_gain_period":   1
        }]
}

Output of Camera Server

Camera data is sent to pipes in a binary form with a metadata struct followed by the raw image data (in a format defined by the metadata):

/**
 * The metadata for the camera image. One of these is sent before every frame
 */
typedef struct camera_image_metadata_t
{
    uint32_t magic_number;  ///< set to CAMERA_MAGIC_NUMBER
    int64_t  timestamp_ns;  ///< timestamp in apps-proc clock-monotonic of beginning of exposure
    int32_t  frame_id;      ///< iterator from 0++ starting from first frame when server starts on boot
    int16_t  width;         ///< image width in pixels
    int16_t  height;        ///< image height in bytes
    int32_t  size_bytes;    ///< size of the image, for stereo this is the size of both L&R together
    int32_t  stride;        ///< bytes per row
    int32_t  exposure_ns;   ///< exposure in microseconds
    int16_t  gain;          ///< ISO gain (100, 200, 400, etc..)
    int16_t  format;        ///< raw8, nv12, etc
    int32_t  reserved;
} __attribute__((packed)) camera_image_metadata_t;

The complete list of image types that can be sent through a camera pipe can be found here.

voxl-inspect-cam output with multiple output types:

voxl-camera-server supports 4 output modes:

Preview Output

The default output for each sensor using voxl-camera-server is the preview channel output (and this is the most common output mode for clients that subscribe to voxl-camera-server). The preview output can be thought of as a low latency minimal resolution stream. For example, voxl-qvio-server takes this output as its input for the tracking sensor, performs VIO processing, and publishes its qvio-overlay in this format via MPA. Config parameters pertinent to this output are:

1. preview_width
   
2. preview_height
   

NOTE: If you are not using the encoded pipe as the channel to voxl-streamer, these are the parameters used to change dimensions of frames for the RTSP stream.

Stream Output

NOTE: This is only enabled (by default) on hires cameras. The name of the pipe this encoded data is being sent to is: hires_stream

voxl-camera-server V1.4.5 and above has the ability to compress image frames using OMX for a no-copy, hardware accelerated and memory optimized data path to publish compressed data. This then can be leveraged by voxl-streamer v0.5.0 and above to input the h264 as input and stream over IP as an RTSP stream. The following config parameters are how a user can leverage this ability in voxl-camera-server:

1. stream_width
   
2. stream_height
   
3. stream_bitrate
   

With the previous config parameters populated, the user can now stream encoded 720p, 2k, and 4K video to voxl-streamer to be viewed in VLC or QGC.

The following video here shows how to setup 720p encoded video stream to voxl-streamer using a VOXL2 and IMX412.

Supported Resolutions

TODO: HOW TO SEE LIST OF VALID RESOLUTIONS FOR ALL CAMERAS AND MODES?

For VOXL 2, there are currently only a subset of three (3) resolutions that work with OMX:

1. 4096x2160
   
2. 2048x1536
   
3. 1024x768
   

The following configurations have been tested

Record Output

Please refer to the voxl-record-video page for how to record output of h264 from the hires_record pipe.

Snapshot Output

The Snapshot output gives the user the ability to take a still picture from whichever stream they are currently viewing through. This ability shows up on the voxl-portal under the stream you are viewing. By selecting the snapshot button on voxl-portal, a still will be saved to disk. The config parameters that are used in this feature are:

1. snapshot_width
2. snapshot_height

Exposure Control

VOXL Camera Server primarily uses ModalAI’s libmodal_exposure for camera auto exposure/gain values, but this can also be changed to use the ISP’s (Image Signal Processing) exp/gain settings, or can be disabled altogether. Additionally, you can use voxl-send-command to manually send commands to update the exposure/gain of a specific camera. If a camera was using auto exposure/gain when such a command is sent, the AE will be disabled until the start_ae command is sent and the camera will sit at the requested exposure/gain.

Manually Setting Exposure and Gain

When an image sensor is configured to use a ModalAI exposure algorithm, auto_exposure_mode: modalai, one can manually set exposure and gain values from the command line. This can be useful for debugging image sensor performance.

voxl-send-command hires set_exp_gain <exposure> <gain>

For example:

voxl-send-command hires set_exp_gain 200 150

Camera sensor options and capabilities

For the available camera configurations as well as the accepted FPS/Dimensions associated to each camera sensor, please follow the link here.

Inspecting

The voxl-inspect-cam tool provides a simple way of inspecting camera frames in-terminal and the source serves as a good example for how to read from camera pipes on an embedded process.

Streaming Options

Stream Over Web/HTTP using VOXL Portal

VOXL Portal provides a simple, easy-to-use web interface for viewing images from camera server or any other service providing an image on VOXL. This can be accessed by hitting the IP address of the modalAI device running the voxl-portal service (which can be enabled with a systemctl enable voxl-portal). Once on this page, you will be able to see the preview stream of whatever camera sensors are plugged into the device.

Stream Over RTSP

The VOXL Streamer application can be used to stream video from voxl-camera-server over RTSP.

Stream to ROS

The VOXL MPA to ROS node can be used to provide data coming out of camera server to ROS.

TODO Troubleshooting

Source

The source code for VOXL Camera Server can be found here