The JPEG Committee has launched a new activity: JPEG Pleno, which is targeting a standard framework for the representation and exchange of new imaging modalities such as light-field, point-cloud and holographic imaging. In addition to the representation it is also a target to define new tools for improved compression while providing advanced functionality support for – but not limited to – image manipulation, metadata, image access and interaction, privacy and security. This paradigm shift caused by computational imaging approaches will be as disruptive for the photography markets as the migration from analog film to digital pictures. Hence, JPEG Pleno investigates how this evolution can be properly addressed while taking into account JPEG’s legacy formats.

Rationale

Tremendous progress has been achieved in the way consumers and professionals capture, store, distribute, display and ultimately use images. We have been witnessing an ever-growing acceleration in creation and usage of images in all sectors, applications, products and services. This widespread and still growing use of images has brought new challenges for which solutions should be found. Among others, image annotation, search and management, imaging security and in particular privacy, efficient image storage, seamless image communication, new imaging modalities, and enhanced imaging experiences are just few examples of challenges to which scientific community, industry, service providers, and entrepreneurs have responded in the past, including continuing improvement of existing solutions and creating new ones.

During the past 25 years, the Joint Photographic Experts Group (JPEG) has been an example of such efforts, and it has offered image coding standards which can cope with some of the above challenges. This work has resulted in a series of successful and widely adopted coding algorithms and file formats such as JPEG and JPEG 2000, and more recently the JPEG XR, JPSearch, JPEG XT, JPEG Systems and JPEG AR families of imaging standards.

JPEG format is today a synonym of modern digital imaging, and one of the most popular and widely used standards in the history. Images created in JPEG format now exceed several billions per day in their number, and most of us can count a couple, if not more, JPEG engines in devices we regularly use in our daily lives; in our mobile phones, in our computers, in our tablets, and of course in our cameras. The JPEG ecosystem is strong and continues an exponential growth for the foreseeable future. In the last two decades, a significant number of small and large companies have been relying on the JPEG format, and this trend will likely continue.

Digital photography markets have known a steady – somewhat linear – evolution over the last decade as it concerns supported resolutions, which was mainly driven by Moore’s law. However, we have reached the era of nanoelectronics and simultaneously we are observing the maturing of micro- and nanophotonics technologies giving rise to an unprecedented and heterogeneous range of new digital imaging devices.

HDR and 3D image sensors, burst-mode cameras, light-field sensing devices, holographic microscopes and advanced MEMS (e.g. DMD and SLM) devices enable new capturing and visualisation perspectives that will finally result in a paradigm shift in the consumption of digital photographic material; i.e. though still static, images will rather be consumed as volumes than planar datasets. This paradigm shift will be as disruptive for the photographic markets as the migration from analog film to digital pictures.

Emerging new imaging modalities

Emerging sensors and cameras will allow for the capture of new and richer forms of data, along with the dimensions of space (e.g. depth), time (including time-lapse), angle and/or wavelength (e.g. multispectral/multichannel imaging). Among these richer forms of data, one can cite point-cloud, light-field or holographic data.

A point-cloud data is a set of data points in a given coordinate system. Such dataset is usually acquired with a 3D scanner or LIDAR and subsequently used to generate and represent a 3D surface. Combined with other sources of data (like light-field data, see hereunder), point-clouds open a wide range of new opportunities for immersive browsing and virtual reality applications.

Light-field data (aka plenoptic data) records the amount of light (the “radiance”) at every point in space, in every direction. This radiance can be approximated and captured by either an array of cameras (resulting in wide baseline light-field data) or by a light-field camera that uses microlenses to sample each individual ray of light that contribute to the final image (resulting in narrow baseline light-field data).

Holographic data currently mainly concerns holographic microscopy modalities that typically produce interferometric data and electro-holographic / computer-generated holographic (CGH) displays that use holographic patterns to reproduce a 3D scene. However, considering the maturing of the underlying technologies that are enabling macroscopic holographic imaging systems, it is to be expected that in a near future also this type of data will be flooding our imaging markets. In terms of functionality, holographic data representations will carry even more information than light-field representations to facilitate interactive content consultation.

These new data types can then be processed to recover additional scene information and to render this information in novel ways.

Functionalities and applications

Among the many new and exciting imaging functionalities provided by these richer sources of data is the ability to manipulate the content after it has been captured; these manipulations may have different purposes, notably artistic, task-based and forensic. For instance, it would be possible for users to change, in real time, focus, field of depth and stereo baseline, as well as the viewer perspective; such media interactions and experiences are not available with conventional imaging formats. Moreover, relighting allows users to change the mood of an image.

Depth or other geometric information about the scene is another very useful component, which could be derived from a light-field, a point-cloud or a hologram. Such information may simplify image compositing and other manipulations such as recoloring. Additionally, accurate 3D scene information could also be used to provide localization within a scene and to provide enhanced capabilities to better detect/recognize objects or actions.

In general, these new forms of imaging allow to overcome the shortcomings of traditional photography, where lack of time or care can cause uncorrectable image artifacts such as out of focus objects, since all capture parameters are set at the time of acquisition. Subsequently, the creation process is largely shifted to the manipulation phase with the shooting itself becoming much less critical and definitive. Depending on the type of data, processing and associated functionalities, a variety of use cases and application domains may gain new and richer dimensions, providing better and more intense imaging-based user experiences. They range from interactive content viewing, cultural environments exploration, and even medical imaging checking to more immersive browsing with novel special effects and nicer or more realistic images. All of this will be possible thanks to a more powerful and creative control of the image creation and manipulation processes.

Representation model

As the acquisition of image data changes to support new applications and functionalities, it is expected that the representation model must also adapt. Since the beginning of digital photography, images have been represented as a two-dimensional array of pixels at a given time instance, where perspective, focus and other aspects of the capture have already been fixed.

In contrast to this existing paradigm, emerging image modalities, such as light-fields, point cloud or holographic formats require a representation format that simultaneously records intensity and colour values at multiple perspectives and distances. The representation of point-clouds will also demand a new format, such that attributes corresponding to each point could be accurately represented. Typical attributes for each point may include colour or intensity, transparency, as well as surface-related data such as normal, and curvature. It should also be noted that in contrast to images, which conventionally have a regular sampling structure on a 2D grid, 3D point-clouds are relatively unstructured. Holographic data requires the representation of holographic patterns (diffraction patterns) that contain 3D scene information as intensity and/or phase over a 2D plane.

As we move towards the development of standards that will support these new representation formats, it will be essential to consider interoperability with widely deployed image formats, such as JPEG and JPEG 2000. Although interaction and manipulation features may be limited, enabling such compatibility would allow any existing media browser or device to view conventional images, e.g., derived from a light-field representation. Similarly, if technically feasible, it would be highly desirable to have a unified format for these new imaging modalities, which could facilitate further exchange and interoperability between the new formats.

Action plan

The JPEG committee intends to interact closely with the actors in conventional and emerging imaging. This will be facilitated by focused workshops that are targeted to understand industry needs in terms of technology and supported functionalities. As a consequence, calls for evidence and/or contributions will be issued to launch new standards or extend existing JPEG standards. The latter consideration is extremely important since JPEG standards have been very successful and wherever feasible the JPEG committee will build upon existing legacy.

Invitation for Participation

One of the strengths of ISO/IEC/ITU-T standards is that they are created by the people that need them. Industry and academic experts drive all aspects of the standard development process, from deciding whether a new standard is needed to defining all the technical content. Getting involved in this process can bring significant advantages to your business and research.

Standards are developed by groups of experts called technical committees. These experts are put forward by ISO’s national members. If you are interested in getting involved, contact your national member. The standardisation will follow IPR policies according ISO/IEC directives. Contact details can be found in the list of national members. For particular information related to JPEG standardization, you can contact as well and/or .

To stay posted on the action plan for JPEG Pleno, please regularly consult our website or optionally subscribe to our e-mail reflector: (in order to subscribe to the mailing list, you simply have to send an empty email to – and – and follow the steps of the e-mail being received).