SONY heralded a new era in 2015 by discontinuing what had until then been pioneering CCD technology. To use an anthropological analogy, analog CCD technology is the Neanderthal of image sensors. Digital CMOS sensors, on the other hand, are the modern Homo sapiens. In their 3rd generation, it is expected they will reach new heights in their evolution.
It was mainly economic reasons that prompted SONY to abandon its CCD production lines and the complex CCD manufacturing process. After all, Darwin’s theory of evolution also applies to image sensors: The strongest species most capable of survival comes out on top. In terms of quality, CMOS sensors have even overtaken some aspects of CCD technology – with low noise behavior, high sensitivity and high image rates, they achieve comparable or better results, even in demanding applications. The favorable production conditions are reflected in a far better price-performance ratio. In addition, CMOS sensors offer a much wider range of functions, lower energy consumption and are more flexible when used in industrial applications.
1. Direct readout method paves the way for CMOS
The evolutionary leap in CMOS sensors lies in the readout mechanism of the pixels. Each area sensor consists of a matrix array with photodiodes that convert the arriving photons into electrons. In CCD sensors, the charges of the individual photodiodes are passed via horizontal and vertical shift registers to a readout amplifier outside the active area. Here, the pixel charges are centrally readout and converted into an analog voltage. By contrast, CMOS technology reads out each pixel directly via transistors positioned over the pixel. The signal is converted via the readout circuit, digitized for low noise and finally transferred in parallel via LVDS (Low-Voltage Differential Signaling) wires.
For a long time, the decentralized readout of CMOS pixels was a disadvantage because the precise readout value at the pixel is dependent upon the physical characteristic of the transistors assigned to the individual pixels. As these always exhibited slight differences in the early days of CMOS, they produced high fixed-pattern noise. The increased background noise also resulted in poor sensitivity.
Another issue was that the transistors were relatively large compared to the pixel size. With 3 transistors needed for a rolling shutter and 5 to 6 transistors for a global shutter, this led to a significant reduction in the active pixel area and thus in luminosity. The large amount of space needed for the transistors was to the detriment of image quality and sensitivity. Therefore, for a long time development was focused on fast rolling shutter sensors. But this is exactly what redefined the evolutionary limits of CMOS technology and to some extent the use of image processing for machine vision. Industrial production and applications in the automotive sector, medical technology and many virtual reality applications need high-quality global shutter sensors without artifacts for fast moving processes in real time and outstanding image quality.
2. CMOS takes the image quality and speed to the next level
The decisive step in the evolution of CMOS was taking images using the global shutter principle and combining this with vastly increased speed and outstanding image quality. SONY first made this breakthrough with the development of its 1st global shutter generation, the IMX174. This established the preconditions for the CMOS age in the industrial sector and the development of image processing into the eye of automation. In contrast to the rolling shutter method, there is no delay between the pixel lines thanks to the simultaneous exposure and emptying of all pixels. Fast-moving objects can be imaged by simultaneous capture with global shutter, without artifacts and with increased sharpness.
Capturing all pixels in parallel is not the only advantage: based on their architecture and the parallel readout of all pixels, CMOS sensors are many times faster than the conventional CCD equivalent. While the target frames per second (fps) rate used to place constraints on the application, CMOS technology permits very high frame rates and it is now the interface that represents the main constraint on maximum speed.
The advancing miniaturization of transistors also enables a much-improved ratio between sensor size, active pixel area (fill factor) and the maximum utilization of the incidental light, i.e. maximization of the ratio between the number of photons striking the sensor and the number that are actually converted. CMOS sensors can now used in low-light applications and produce very good image quality.
Another important advantage of CMOS technology can be found at extremely high light levels, for example, in direct sunlight or in reflections. CCD pixels emit excess converted electrons to their neighboring pixels (overflow) or pass them on even after exposure has taken place. This produces blooming or smearing effects, which have a negative impact on image recognition. CMOS sensors, with their direct pixel readout, preclude these effects. The earlier disadvantage of high background noise has also disappeared and CMOS sensors now achieve outstandingly low dark noise of up to 1–2 e-, i.e. a very low dark current and high pixel homogeneity.
Other technical innovations, such as the backside illumination of the chip and microlens arrays mounted on the chip to carry more incidental light, allow CMOS sensors to achieve better sensitivities than CCD sensors. The ratio of pixel saturation to sensitivity gives the dynamic range of a sensor, and CMOS sensors now surpass CCD technology in this area with values of up to 80 dB. High sensitivity in combination with greater speed produces a large performance increase in image processing applications with CMOS sensors.
3. The third generation of CMOS brings new benefits
Sony achieved further pixel architecture improvements in its 2nd global shutter generation and implemented additional functionality of relevance to machine vision applications. Its 3rd generation, however, will have one important innovation above all others: Sony’s SLVS-EC (Scalable Low-Voltage Signaling with Embedded Clock) which introduces a new sensor interface that takes account the constantly increasing requirements for resolution and speed. According to market insiders, the new sensors can achieve a doubled bandwidth compared to the Sub-LVDS based sensors of the 2nd generation.
& SLVS 8CH
The 1st generation SONY CMOS Global Shutter set an initial benchmark with the IMX174, which enjoyed a great response in the image processing market. Suddenly, it was possible to implement applications with an image quality and speed that was inconceivable with CCD technology. With a pixel area of 5.86 µm, a very high saturation of 30,000e- was already reached in the first generation, increasing the dynamic range to 75 dB, “despite” the still high readout noise of 5e-.
In its 2nd generation, SONY shifted the focus to the requirements of Machine Vision. The additional bit depth of 8 bits reduced bandwidth requirements, while twice the number of channels doubled the output speed to 9.5 Gbps. Extended functionalities such as additional trigger modes were also integrated. The reduced pixel size of 3.45 µm reduced the saturation to 11,000e-, while the significantly lower readout noise of 2e- in the 2nd generation CMOS global shutter retained a high dynamic value of 74 dB.
The innovations of the 3rd generation will deliver clear improvements, mainly in terms of image quality and speed. Based on these improvements, an increase in performance is seen through improved detection quality in applications with moving objects such as running production lines and robotics applications as well as in the ITS and automotive sectors. The increase in pixel size will produce a much higher saturation than the second generation and might nearly achieve the value of the first generation. Combined with the expected low readout noise, the maximum dynamic range will reach a new peak, making improved light-dark detection feasible even in difficult lighting conditions. Based on the different quality improvements, it will be no longer possible to transfer this increase in image data any faster using the existing standard interfaces. SONY has therefore developed the SLVS-EC standard with 8 channels, which is expected to double the maximum output speed compared to the 2nd generation.
4. Successful CMOS implementation needs detailed knowledge
With their vastly improved image qualities, speeds and expanded functionalities, modern CMOS sensors have developed into the fundamental digital component of any vision system. They can be relatively easily designed and implemented as integrated circuits. However, the application challenge lies less in the specific implementation rather than in achieving the best-possible image quality. Developers and engineers can use evaluation boards and matching Reference Design Kits (RDKs) to select the correct sensor and utilize its full functionality with its detail settings. RDKs including Gerber files, sensor boards with prefabricated designs and finished IP blocks allow you to work directly on the sensor and prove its suitability for the desired application and the required image quality.
Companies that develop their own design can avoid standard errors, shorten development cycles and thus achieve faster implementation and time-to-market by employing the ready-to-go IP packages provided by image processing experts. Manufacturers or OEMs whose core competence does not lie in image processing or camera technology can retain their original focus with the assistance of external experts and still implement mature imaging solutions in their devices. In addition, image processing experts can provide support for the selection of all other system components, such as suitable lenses, and therefore put together a perfectly harmonized package.
5. Higher CMOS image quality and speed driving technical progress
Properly matched components are a basic requirement for the success of “embedded” image processing in the field of industrial digitization. With the increasing performance of processors and sensors and easy availability, image processing now has a much greater capacity for analyzing and interpreting images more effectively.
In Industry 4.0, artificial intelligence with deep learning approaches supports total automation as well as autonomous, robot-controlled production lines. In the consumer segment, autonomous cars are learning to see, and virtual reality glasses transport wearers to new worlds. CMOS sensors are making a significant contribution to this development. In classic inspection tasks, the design of the CMOS sensor reduces the risk of false measurements or incorrect detection where soldering and assembly tolerances are low.
Higher image quality and higher speed potential of CMOS sensors support demanding electronics and medical technology applications that require a high level of precision. In medical technology, for example, it’s possible to implement applications such as “lab-on-the-chip” or digital X-rays which accelerate imaging and diagnostics for the detection and treatment of illnesses.
CMOS technology allows cameras a higher frame rate. Combined with the right lighting, this means shorter exposure times, resulting in fundamental image improvements as well as sharper, contrast-richer images, all thanks to the CMOS architecture. Detailed image detection also reduces post-processing time. Shadows, blooming effects and image distortions are improved or even avoided altogether when taking the shot, due to the pixel structure and the extended dynamic range of the CMOS sensors.
Their speed allows CMOS sensors to compile several shots in HDR mode and improve image quality with an increased bit depth and finely graded colors, with no loss of speed. Visible patterns can be detected faster and more effectively on X-ray or microscope images, and findings can be dealt with and documented digitally. CMOS sensors can also deliver more effective imaging and image evaluation at high speeds in outdoor and ITS (Intelligent Traffic Surveillance) applications. This enables more vehicles to be captured in moving traffic, while changing light conditions caused by weather or shadows have less influence.
6. Miniaturization and multidimensionality with CMOS
Compared to CCD sensors, CMOS sensors consume much less energy and therefore generate less heat. Among other things, this results in reduced noise behavior. One of the biggest benefits of CMOS is with the on-chip-technology. This means less electronic components are required allowing more electronic components to be installed in a smaller space due to the low heat output. Camera modules with CMOS sensors therefore need less space and are the sensors of choice for hand-held and mobile devices.
CMOS is the new standard and state-of-the-art technology. All new developments are now using CMOS technology. The Neanderthal CCD has pretty much died out in this area, relegated in practice to supporting a few selected special applications. Where CMOS sensors show their advantages is in the development and production of new 3D and 4D imaging processes. For stereo measurements, for example, advancing miniaturization combined with high resolution and speed means there is no longer any need to install 2 cameras. CMOS technology makes it much easier to realize a space-saving combination of 2 sensors feeding into one common camera board and enabling a compact stereo camera design to take 3D shots by means of triangulation. Above all, the ability to integrate logic blocks onto CMOS sensors facilitates the development of advanced and innovative image processing methods. Time-of-flight technology, which works as a processing unit on-the-chip with time filters to measure distance, and 4D event cameras, which use on-board processing to detect changes in images over time, would not be feasible with the basic functionalities and structure of an analog CCD sensor.
7. CMOS supports fully integrated automation with embedded vision
In combination with the general miniaturization of sensors and electronic components, CMOS technology has advanced to become the driver of image processing. The aim for the future is increased speed, higher resolution and above all a greater degree of integration of CMOS sensors into image processing systems. Sensors will need even less space as image quality and speed increase. The market leaders SONY and ON Semiconductor as well as providers such as e2V are working on integrating additional functionalities directly onto the sensor. The less processing required downstream, the faster the application can retrieve finished image data and control systems and processes with less effort. The power requirement and the heat generated will continue to decrease.
As a result, CMOS technology with its technical advantages and advances is a driver for Embedded Vision, new industrial applications and the integration of image processing into robotics and automation, the Smart Factory and the digitized processes of Industry 4.0. Developers and users can look forward to seeing which technical innovations manufacturers will come up with to begin the next stage of evolution.
The FRAMOS Sensor Tech Days on April 25 and 26 in Munich will take an in-depth technical look at the current state of CMOS technology and coming roadmaps. At the information and dialog platform for camera developers and R&D engineers, SONY Japan will be presenting its global shutter portfolio and new SLVS-EC high-speed interface in detail. The FRAMOS development engineers will also explain the matching SLVS-EC Reference Design Kit with FPGA implementation and IP. A further SONY presentation will showcase the 3rd generation CMOS global shutter, including a special live demonstration. e2v will be presenting its new EMERALD series. The new product family is based on the world’s smallest global shutter pixel of 2.8 µm. ON Semiconductor will also be presenting their current GS portfolio and roadmap.