MIPI CSI-2 Camera Interface for Next-Generation Embedded Apps
Perhaps more than any other camera interface, MIPI Camera Serial Interface 2 (MIPI CSI-2) provides point-to-point image and video transmission for mobile and embedded vision applications. Developed by the MIPI Alliance (www.mipi.org), MIPI CSI-2 defines an interface between an image sensor module and a host processor such as a System on Chip (SoC) and suits single or multi-sensor embedded vision applications. MIPI CSI-2 provides end-to-end conduit solution between image sensor modules and an SoC for a broad range of product platforms including mobile, client, Internet of Things, and automotive.
MIPI imaging solutions for machine vision applications build on the CSI-2 imaging conduit infrastructure developed for mobile product platforms. MIPI Camera Working Group (CWG) member companies leverage existing CSI-2 infrastructure for near-real-time perception and decision making applications like latency reduction and transport efficiency, smart region of interest for inferencing, and unified serial link with replay protection and fast BTA (bus turn around) to dramatically reduce the conduit wires. In terms of flexibility, MIPI CSI-2 enables cross pollination of IP solutions and reusability of engineering efforts across multiple systems, explains Haran Thanigasalam, Chair, MIPI Alliance Camera Working Group.
“The CWG continues to further advance solutions for next-gen machine vision capabilities like functional safety aligned with ISO-262262:2018, security, and ultra-low power, always-on sentinel conduit (AOSC) solution for enhanced always-on machine vision and surrounding awareness needs,” he says. “In addition, CWG has also developed complementary Camera Command Set (CCS) for CSI-2, which enables the development of unified imaging drivers for rapid bring-up and deployment of image sensor modules on complex machine vision platforms.”
Related: Embedded deep learning creates new possibilities across disparate industries
Since its introduction in 2013, MIPI CSI-2 has been implemented in an ever broader range of applications, and several companies now develop board-level and modular cameras with the interface to suit the needs of integrators developing space-constrained, low-cost, low-power OEM and embedded vision systems. Allied Vision (Stadtroda, Germany; www.alliedvision.com), for example, manufactures the Alvium series of cameras with a proprietary application-specific integrated circuit (ASIC) for embedded vision processing. Supporting OpenCV and available with MIPI CSI-2 and USB3 Vision interfaces, the cameras come in bare board (Figure 1), open housing, and closed housing form factors with various lens mount options, giving developers design flexibility, yet share a common driver to ease integration, while offering on-board image processing, intelligent power management, and a cost-optimized design, according to the company.
MIPI CSI-2 interface-based Alvium 1500 C series cameras feature CMOS image sensors ranging from the 0.5 MPixel (116 fps) PYTHON 480 up to the 5 MPixel AR0521 (67 fps) from ON Semiconductor (Phoenix, AZ, USA; www.onsemi.com). Alvium 1800 C series with MIPI CSI-2 interface make use of sensors like the 0.4 MPixel IMX287 (280 fps) CMOS sensor from Sony (Tokyo, Japan; www.sony.com) up to the 19.7 MPixel IMX183 (25 fps), also from Sony.
“The CSI-2 backend in our Alvium camera series aims to make integration of such cameras in demanding embedded applications easier and quicker by providing a standard set of image correction and pre-processing functionalities independent of the sensor,” says Gion-Pitschen Gross, Product Manager. The full functionality is available through our open source drivers so system designers can focus on their algorithms and don’t have to spend time integrating the basic camera functionality.”
Camera manufacturer Basler (Ahrensburg, Germany; www.baslerweb.com) designs embedded vision products compatible with a variety of SoCs. The dart BCON for MIPI camera modules are now compatible with NXP Semiconductor’s (Eindhoven, Netherlands; www.nxp.com) i.MX 8 applications processors. The company also further expanded its partnership with NVIDIA (Santa Clara, CA, USA; www.nvidia.com). By integrating the dart BCON for MIPI camera modules on the Jetson platform (Nano or TX2, for example), developers can create AI-enabled embedded vision applications.
In its embedded vision cameras, Basler (Ahrensburg, Germany; www.baslerweb.com), offers a proprietary BCON for MIPI interface designed specifically for systems with MIPI CSI-2 connections. The company developed these products, according to Dr. Thomas Rademacher, Product Manager for the embedded vision camera modules, as a compact and cost-saving system to replace traditional smart cameras.
The company’s dart cameras with BCON for MIPI interface (Figure 2) offer resolutions ranging from 5 to 13 Mpixel (5, 8 und 13) – with an S-Mount or with no mount. These cameras were developed, according to Dr. Thomas Rademacher, Product Manager for the embedded vision camera modules, as a compact and cost-saving system to replace traditional smart cameras. Image pre-processing occurs either in the image signal processor (ISP) of the respective host system-on-chip (SoC) e.g. the new NXP i.MX 8M Plus or the Qualcomm Snapdragon (San Diego, CA, USA, www.qualcomm.com) or directly on the camera. Models with in-camera image pre-processing based on the BCON for MIPI interface include four camera modules (color with bare board-level or S-Mount options) with either 5 MPixel (60 fps) or 13 MPixel (30 fps) CMOS image sensors from ON Semiconductor. As a processor specific driver is needed, these cameras come with a ready-to-use driver package.
To make the starting point for developers easy, the company also develops a variety of BCON for MIPI-based embedded vision kits for NXP, NVIDIA, and Qualcomm processors. These assembly kits comprise all the important components for evaluating Basler dart camera modules and the respective processor platform for ease of design into embedded vision systems.
Related: CPUs, GPUs, FPGAs: How to choose the best method for your machine vision application
"Our goal is to create the optimal embedded vision solution for our customers. The Basler dart BCON for MIPI camera modules utilize the processing capabilities of a variety of SoCs to enable our customers to achieve lean systems without compromising image quality," explains Gerrit Fischer, Head of Marketing Module Business at Basler. "This makes the dart BCON for MIPI an ideal camera for demanding embedded vision systems and IoT applications.”
D3 Engineering (Rochester, NY, USA; www.d3engineering.com) makes board-level sensor modules with direct digital sensor output in MIPI-CSI2, LVDS, and parallel video. These D3SM modules target integration into embedded systems, board-level connector, and lens mount with an actively aligned lens. D3SM modules support popular development boards such as the D3 DesignCore Camera Mezzanine Board, the Arrow Electronics (Centennial, CO, USA; www.arrow.com) DragonBoard Camera Kit, and the BeagleBoard from Texas Instruments (Dallas, TX, USA; www.ti.com).
In its camera boards for embedded vision models, e-con Systems (San Jose, CA, USA; www.e-consystems.com) offers 18 different options based on MIPI interface. Cameras range from 2 MPixel to 13 MPixel CMOS image sensors in color, monochrome, NIR, and raw Bayer. Capable of frame rates ranging from 30 to 120 fps, these cameras are designed for use with such embedded vision development kits as NVIDIA’s Jetson Nano (Figure 3), Xavier, TX1, and TX2, as well as Connect Tech’s (Guelph, ON, Canada; www.connecttech.com) Rogue Carrier board (AGX101), Variscite’s (Lod, Israel; www.variscite.com) DART-MX8M-MINI, Rockchip’s (Fuzhou, China; www.rock-chips.com/a/en/) RK3399, and Google’s (Mountain View, CA, USA; www.google.com) Coral development board.
Additionally, e-con Systems makes NileCAM cameras, NileCAM is a Series of cameras are based on GMSL SerDes technology. These cameras can support 3 or 15-meter coaxial cable. The NileCAM30 camera is based on 3.4MP AR0330 CMOS image sensor from ON Semiconductor. This camera can support both MIPI and USB interface.
Enabling 3D Time of Flight (ToF) capabilities, the Helios Flex camera module (Figure 4) from LUCID Vision Labs (Richmond, BC, Canada; www.thinklucid.com) features the new 0.3 MPixel Sony DepthSense IMX556 CMOS image sensor. Connecting to the camera’s MIPI interface to NVIDIA’s Jetson TX2 embedded board, the Helios Flex offloads raw data for CUDA-accelerated processing on the graphics processing unit (GPU) of the Jetson TX2. By using the camera expansion MIPI port, developers and integrators can connect additional devices and sensors to the USB3 and Ethernet ports. This camera provides 3D ToF images at 30 fps and is available in a bundle consisting of a camera module, camera adapter board, cables, and AC power supply—with NVIDIA’s Jetson TX2 sold separately.
Available in board and cased versions, Omron Sentech’s (Hoffman Estates, IL, USA; www.automation.omron.com) MIPI cameras use the 1.3 MPixel Sony ISX017 STARVIS CMOS image sensor. The color cameras reach a frame rate of 60 fps and feature multiple lens mount options.
Designed for direct access to NVIDIA’s Jetson Nano, TX2, and AGX Xavier, as well as the Raspberry Pi, board-level MIPI CSI-2 cameras (Figure 5) from The Imaging Source (Bremen, Germany; www.theimagingsource.com) come in color and monochrome versions. Based on Pregius and STARVIS CMOS image sensors from Sony ranging from 0.3 MPixels to 8.3 MPixels, the monochrome cameras reach frame rates of up to 240 fps while the color versions feature the same sensor sizes and achieve up to 120 fps. Both color and monochrome cameras can used right away with HALCON machine vision software from MVTec (Munich, Germany; www.mvtec.com) Additionally, the company offers housed and board-level cameras with FPD-Link III bridge—a MIPI extension that overcomes MIPI’s short connection distance and extends cable lengths up to 15 m.
Related: Low-cost development tools reduce the cost of embedded vision
“Modern cameras and vision sensors using the MIPI CSI-2 interface, coupled with NVIDIA’s fully-scalable Jetson platform deliver a much-reduced footprint and cost savings as compared to non-embedded computer-based systems,” says Roland Ackermann, Sales Director EMEA. “The MIPI interface standard also simplifies the integration of embedded systems by giving developers a common software approach.”
“However,” continues Ackermann, “There are still hurdles to overcome. Each camera sensor needs its own kernel mode driver and ISP configuration—know-how that The Imaging Source offers its customers in maximizing the potential of their embedded vision applications.”
Compatible with CPU boards from NVIDIA, Raspberry Pi, Rockchip, Arrow, NXP Semiconductors, and several others, Vision Components (Ettlingen, Germany; www.vision-components.com) manufactures several different MIPI-based cameras in color or monochrome versions (Figure 6).
Most VC MIPI modules use Sony Pregius or STARVIS sensors, but two cameras have sensors from OmniVision (Santa Clara, CA, USA; www.ovt.com). Sensor sizes range from 0.3 MPixels to 20.2 MPixels and frame rates from 24 fps to 530 fps.
“The MIPI interface comes from the consumer market and thus provides a large technology offering due to its size,” says Jan-Erik Schmitt, Vice President of Sales. “Using the MIPI CSI-2 interface, the VC MIPI camera modules can be used with a wide range of available and appropriate embedded processor architectures for any application imaginable.”
He adds, “As a result of MIPI’s high flexibility, developers can also achieve extremely short time-to-market and quickly get results based on the two main components needed to create an embedded vision solution—the image acquisition device and the processing unit—while keeping the costs down from the beginning.”
Looking toward the future, MIPI CWG member companies are at work on evolving the CSI-2 interface to dramatically improve human-to-human collaboration and alleviate human-machine friction within machine vision, according to Thanigasalam.
Companies mentioned
Allied Vision
Stadtroda, Germany
www.alliedvision.com
Arrow Electronics
Centennial, CO, USA
www.arrow.com
Basler
Ahrensburg, Germany
www.baslerweb.com
Connect Tech
Guelph, ON, Canada
www.connecttech.com
D3 Engineering
Rochester, NY, USA
www.d3engineering.com
e-con Systems
San Jose, CA, USA
www.e-consystems.com
Google
Mountain View, CA, USA
www.google.com
LUCID Vision Labs
Richmond, BC, Canada
www.thinklucid.com
MVTec
Munich, Germany
www.mvtec.com
NXP Semiconductors
Eindhoven, Netherlands
www.nxp.com
NVIDIA
Santa Clara, CA, USA
www.nvidia.com
OmniVision
Santa Clara, CA, USA
www.ovt.com
Omron Sentech
Hoffman Estates, IL, USA
https://automation.omron.com
ON Semiconductor
Phoenix, AZ, USA
www.onsemi.com
Qualcomm
San Diego, CA, USA
www.qualcomm.com
Rockchip
Fuzhou, China
www.rock-chips.com/a/en/
Sony
Tokyo, Japan
www.sony.com
Teledyne e2v
Chelmsford, Essex, UK
www.teledyne-e2v.com
The Imaging Source
Bremen, Germany
www.theimagingsource.com
Variscite
Lod, Israel
www.variscite.com
Vision Components
Ettlingen, Germany
www.vision-components.com
