Applications for JPEG 2000
Digital Cinema describes the use of movies with a digital data representation in best quality. Traditionally movies are shot on film and projected with film. Today, this is done with digital cameras and digital projectors. Because of the huge amount of data within this application area data compression is necessary. In contrast to Electronic Cinema, which uses the digitization of the film for new commercialization pathes, Digital Cinema replaces only the film chain from the acquisition to the film theatres. Therefore Digital Cinema must achieve and surpass the traditional best film quality. The parameters for the digital representation of the movie have to be much more extensive than in standard videos.
Other compression standards have different limits for the use in Digital Cinema. This can be the maximum resolution, the compression possibilities (only lossy), the sampling type, the colorspace or the bit depth. However, JPEG2000 is an excellent compression standard for the use in Digital Cinema, because it delivers enough headroom in the description of digital movie data and has outstanding features, which can be used. Some features of JPEG2000 are intraframe coding for a simple editing access, lossless compression capabilities, metadata insertion, scalability in resolution and quality and so on. All features of the still Image JPEG2000 Standard 15444 - Part1 can also be used.
Requirements in data compression for Digital Cinema including high dynamic range, different color spaces, highest image resolutions, best compression quality upto lossless compression, and so were made possible by using JPEG 2000.
JPEG 2000 has been adopted by the broadcast industry as mezzanine compression in the live production workflows. The compression offers unique benefits suitable for video production as alternative to uncompressed video. Today JPEG 2000 is used for its high quality and low latency in video over IP applications such as Contribution Links (live events to studio transmission) and recent IP-based broadcast studio infrastructures. Moreover, it is also used as the master format for content storage. The broadcast implementations essentially rely on the JPEG 2000 Amd3 for Broadcast Contribution and the JPEG 2000 Amd8 for Interoperable Master Formats (IMF).
In 2014, several companies received an Emmy® Award for breakthroughs in the Standardization and Productization of transporting video with JPEG 2000 Broadcast Profile in MPEG-2 TS over IP Networks.
Several main benefits for JPEG 2000-based broadcast production workflows exists:
- Intra-frame compression JPEG 2000 is an intra-frame based encoding scheme, as it encodes each frame independently. This is a great advantage for content editing applications, as the video signal can be cut at any place without repercussion.
- Video quality JPEG 2000 offers an enhanced tool-set that is especially well-suited for high-quality video. While the driving force for MPEG has always been to deliver good video quality at low bitrates, e.g. for TV/Mobile/Internet broadcasting by improving the temporal prediction, the driving force behind JPEG 2000, on the other hand, has been to produce a standard with high-quality compression of single pictures or frames. JPEG 2000 can allocate more bits per sample, e.g. 10-bit video is offered, which is in line with broadcast production specifications and regular studio practice.
- Less visual artefacts Bit-errors in a JPEG 2000 stream create less visual artefacts than MPEG solutions, as the error appear as a short-lived blur in the picture much less visually disturbing than the blocking effects which can also be much longer lasting.
- Low latency Low latency processing is crucial for live TV contribution. JPEG 2000 technology is able to provide this required ultra low latency as there is no dependency between the signaled frames. In general, the technology is able to easily achieve encoding and decoding latencies of less than 1.5 frames, and some implementations can even offer latencies of less than one frame.
- Symmetrical complexity and cost structure for real-time video It takes the same horsepower to do a compression and a decompression and the same amount to do any quality of compression. The architecture of broadcast production systems has the same number of transmitters/encoders and receivers/decoders. The cost and complexity of the transmitter and the receiver should therefore preferably be the same, which is the case with JPEG 2000. Encoding and decoding can be done using the same chips.
- Robustness transmission JPEG 2000 is a particularly good choice for contribution of video over IP as it is exceptionally robust to transmission errors. As described earlier, JPEG 2000 provides no blocking artefacts and any errors that are introduced are more pleasant to the eye. Additionally, because JPEG 2000 has no error propagation between frames, artefacts are much shorter-lived. Intrinsically, JPEG 2000 also has special mechanisms to increase the robustness to transmission errors.
- Robustness to multiple encoding In a chain of successive compression-decompression processes over the same original material, JPEG 2000 compression technology sustains the same quality very well and it is robust to pixel shifts.
- Easy editing thanks to scalability in both resolutions and quality The resolution scalability enables editors to manipulate easily a sequence by manipulating only a low resolution (proxy) of the movie. All the operations done on the low resolution are then applied on the full resolution version.
Image archives and databases
One early use of JPEG 2000 will be as a base file format in image archives and databases. Traditionally, image archives store multiple copies of an individual files at varying resolutions and quality levels so that they can supply appropriate image data on request. In addition, considerable metadata is held about each image to allow it to be easily classified and retrieved.
JPEG 2000 files typically can have extensive metadata stored with them, in a standard compliant XML environment. As well as allowing selected metadata from an image database to be distributed to its users, this does permit interchange of image files with metadata between databases, and removes the need for an extensive manual data entry stage when cataloguing new images. In addition, the files can be stored at high quality in a lossless, colour managed environment, with conversion to lower resolution or lower quality performed 'on the fly'. The ability of part of a JPEG 2000 file to be used for generation of such modified images also means that it becomes practical to provide other capabilities on demand.
One example might be to watermark each image as delivered, not only with details which communicate authorship or ownership, but also transactional information. This could include licensing restrictions, details of the customer, or information which would allow the image to be easily recognised through some automated process designed to test for breaches of copyright.
The new Part 8 of the JPEG 2000 standard (JPSEC) dealing with security addresses these possibilities, whilst Part 9 (JPIP) defines how interactive applications between a client and server can be created. This too will be very important in the image database arena - as examples it makes retrieval of selected parts of an image much faster and easier to control, permitting 'pan and zoom' operations on part of an image. Demonstrations of this technology already exist (for example using Kakadu) in which several areas of an image can be selected by a user and are delivered more rapidly that the remaining less interesting parts. A range of novel browsing opportunities exist therefore for remote client software, making the delivery of large high quality image information under user control a practical reality.
JPEG 2000 has many characteristics which are useful to one of its target market areas, medical imaging. Some background to this has been covered in a JPEG committee document (N2782) which also gives some useful information about how JPEG 2000 works. One key aspect which often concerns the medical profession is the need to ensure that images can be communicated losslessly, without any distortions introduced by a compression process that may lead to mis-diagnosis. This often results in huge files, which can be difficult to store, handle, and communicate. JPEG 2000 can be used to encode files completely (or partially) losslessly, and provides good compression performance for this purpose (similar, for example, to that offered by JPEG's optimised method for such compression, JPEG-LS (IS 14495)). It does however have several additional features which make JPEG 2000 particularly attractive for medical imaging:
- volumetric image support through Part 10
- selected parts of the image can be defined as Regions of Interest - they can then be delivered before other parts of the image, or losslessly, whilst other parts of the image that are less critical use normal lossy compression.
- the JPEG 2000 codestream can be ordered to deliver images of lower resolution, or reduced quality, well before the full image can be transmitted. This helps significantly in browsing applications, and means that only one file is needed for several applications
- extensive metadata can be included with the image, in a tight association. This means that files can be transmitted between recipients which can easily be processed, or indexed into an existing database. Some applications, such as those relating to DICOM standards, have their own sophisticated methods for handling this metadata, and JPEG are working with the DICOM committee to ensure that these two important standards can be easily integrated
- many different forms of image can be usefully compressed using JPEG 2000 - for example radiological, MRI, CAT and other medical imaging modalities, which use non-visual sensors, and may use enhancement techniques such as pseudo-colouring the resulting image
Many cultural heritage institutions such as Museums and Art Galleries have very extensive collections which are not visible to the public because of display capacity and other reasons. Projects, such as 'NOF-Digitise' in the UK with a budget in excess of $80M, are being created to try and provide on-line learning resources and other solutions for global access. Natural disasters such as fire, earthquake and flooding, as well as the problems created by war, vandalism and terrorism show the need to preserve this information in as accurate a form as possible lest the heritage be lost forever. In addition, it is critically important to use widely adopted standards that have some chance of longevity in the face of technological change. The UK 'Domesday' project, for example, in which the BBC helped many schools and individuals put together a comprehensive record of the UK (in November 1986) to celebrate the 900th anniversary of the Domesday Book was created using a BBC Master computer with a proprietary interface to an analogue videodisc. The problems less than 20 years later are self-evident, and the large expenditure on the project stands to be lost, unless emulators for the material can be created.
The original JPEG standard has been existence almost as long as the Domesday project videodiscs described above. Whilst there are only one or two working Domesday players, and some attempts to rescue the situation, there are hundreds of millions of devices which can image JPEG files. The JPEG 2000 standard has been designed from the ground up to try and address many of the areas which concern the users in the Cultural heritage sector. These include:
- high quality lossless compression with full colour management
- extensions to moving and 3D images with the same advantages
- a royalty and license fee free baseline for widescale deployment
- protection of moral and copyrights through a well defined security mechanism which can (for example) offer undistorted thumbnail viewing and encrypted high resolution viewing from the same image file
- a comprehensive client / server architecture for users to be able to zoom in and out on images, or to request regions of interest to be served ahead of background material
- extensive metadata possibilities, including the incorporation of digital camera information (such as is held in EXIF files), as well as the Dublin Core metadata used as a basis for many cultural heritage projects
- adherence to well accepted and defined standards, including the use of XML, HTTP and others within the defined architecture
- a track record of accuracy and ecceptance by the industry
Wireless communications offers its own particular set of problems. More specifically, wireless networks are characterized by the frequent occurrence of transmission errors along with a low bandwidth. Hence, they put strong constraints on the transmission of digital images. Since JPEG2000 provides high compression efficiency, it is a good candidate for wireless multimedia applications. Moreover, due to its high scalability, JPEG2000 enables a wide range of Quality of Service (QoS) strategies for network operators.
To be widely adopted for wireless multimedia applications, JPEG 2000 has to be robust to transmission errors. To address this issue, the JPEG committee has established a new work item, JPEG 2000 Wireless (JPWL), as Part 11 of the standard. Its purpose is to standardise tools and methods to achieve the efficient transmission of JPEG 2000 imagery over an error-prone wireless network.
The main functionality of the JPWL system is to protect the codestream against transmission errors. More precisely, the protection technique modifies the codestream to make it more resilient to errors, e.g. by adding redundancy or interleaving the data. The decoding process detects the occurrence of errors and corrects them whenever possible.
A second functionality is to describe the degree of sensitivity of different parts of the codestream to transmission errors. This information can subsequently be used for unequal error protection. More specifically, sensitive parts of the codestream can be more heavily protected than less sensitive parts.
A third functionality is to describe the locations of residual errors in the codestream. This information can subsequently be used to make a decoder aware of the information loss and to prevent decoding corrupted parts of the stream.
Using the technologies standardised in JPWL, JPEG2000 becomes very resilient to transmission errors. Therefore, JPEG2000 is an ideal candidate for the efficient transmission of digital images and video in wireless applications. Indeed, recent studies have shown that Motion JPEG2000 is very well suited for video transmission over wireless channels. Specifically, it has been shown that Motion JPEG2000 outperforms the state-of-the-art MPEG-4 in terms of coding efficiency, error resilience, complexity, scalability and coding delay.
While the proposed solutions are not tuned to a specific network protocol, particular attention has been paid to three important use cases: 3rd generation wireless phone networks (3GPP/3GPP2), WLAN (IEEE 802.11 family of standards) and Digital Radio Mondiale (DRM).
Among potential killer applications for JPWL, Multimedia Messaging Service (MMS) knows a very rapid growth and is widely seen has one of the only bright spot in the wireless telecom industry. Other potential applications include video streaming and video conferencing.
Pre-press is the process used when digital files are prepared for printing. Two key requirements of this process are fidelity and consistency. In the past, the pre-press industry has depended on lossless image compression (for example using EPS or TIFF file formats) and colour calibration of all components in the process, using defined lighting and viewing conditions in order to achieve optimum results.
JPEG 2000 offers opportunities to the pre-press industry to both substitute its traditional formats with the more advanced aspects inherent in JPEG 2000, and to re-purpose its content to allow it to be used in Internet publishing or other contexts. The same JPEG 2000 image can generate thumbnails, screen images and print ready material simply by truncating a prepared codestream at different points. In addition, the powerful metadata handling and association in JPEG 2000 files means that Digital Asset Management or workflow processes can be easily linked into pre-press delivery, providing security to the photographer, image creator, and printer alike.
A key facet is the ability of JPEG 2000 to deliver true lossless compression - in one possible operational mode even the colour transform from an defined colour profile such as sRGB is lossless. The ability exists within JPEG 2000 to use a number of well defined colour management profiles, and in particular ICC colour profiles, supporting the CMYK spaces used within the pre-press industry. As the file format can include full colour space definitions (at least in the extended version of JPEG 2000 defined in Part 2 of the standard), including the formula used for transforming to another colour space, proprietary and accurate colour representations can be transferred between systems (at least within the limitations of output devices to render them).
Remote sensing and GIS
Geographic Information Systems (GIS) allow the viewing and analysis of multiple layers of spatially related information associated with a geographical location or region. GIS enables companies and governments to easily analyze the development, maintenance, and impact of roads, vegetation, utilities (water, electrical, communication, sewage).
GIS includes maps, vector information, and imagery. The collection of imagery is commonly achieved through remote sensing. Remote sensing started with aerial photography in the late 1800's onboard a balloon. Airplanes were used to collect information from above in the early 1900's and the first image taken from space was aboard the Apollo spacecraft in 1969. In the early 1970s the first imaging satellite (ERTS-1) collected imagery of the Earth. Images continue to be collected form both space and aircraft and are available for commercial and personal use on the Internet. The challenge for Remote Sensing images for GIS and other applications is the size of the image. Currently, it is common to have images that are greater than 10,000 by 10,000 pixels, multiple bands, and greater than 8 bits per pixel per band. While JPEG DCT is currently being used for the collection, storage and delivery of several GIS applications and remote sensing systems, other compression and file format techniques have become popular because of greater efficiency of storing and accessing large images.
The original requirements for JPEG 2000 included requirements from the remote sensing and GIS community, which have been meet. Greater bit depths, tiles, resolution progression, quality progression, and fast access to spatial locations all contribute to the capability and functionality of JPEG 2000, which make it an ideal technology for the remote sensing and GIS applications. As an open standard, it is expected that JPEG 2000 will become more prevalent in the remote sensing and GIS applications.
Photography has changed the way people record and remember images, events, and scientific information. Photography, which started in the mid 1800's, has continued to evolve over the last two centuries of scientific discovery. The development of flexible film in the late 1800s, color photography in the mid 1900s, and automatic cameras in the late 1900's changed the way photographs are taken and presented. The most recent addition of digital photography has also changed the way people collect, store, modify, disseminate and display images. Digital photography started with the advent of the first commercial digital cameras for consumers and professionals in the early 1990's, along with the first systems for digitizing film images. As the technology advanced, the cost of digital cameras and film digitization services has dropped, and the image quality has increased. The image size for professional portable digital cameras continues to grow, from about 1 Megapixel in 1993 to 10 Megapixels or more in 2003.
As digital cameras evolve, the requirements for the file format used to store the image data continue to evolve also. Digital cameras continue to increase the size and bit depth collected for an image to increase the resolution and extend the dynamic range and color gamut. Digital Photography requires the ability to compress three-band imagery of 8-to-16 bits per component. Digital photography requires efficient, high quality compression as well as rapid decoding of properly sized images for the camera's display screen. Metadata for the proper use and display of the image is a requirement for digital photography.
Scientific and Industrial
Many applications in the scientific and industrial sector are now turning to the use of image material to either replace or enhance existing data records. Examples include the use of satellite or aerial photography imagery to link to a mapping or GIS system, and the ease of use of digital cameras to provide evidence of satisfactory work completion - for example in road pipework excavation. To some extent these depend on the ubiquity and availability of the standard, and hence projects often use well tried and tested solutions. The widespread availability of digital cameras, and PC sofware to display and print the results are likely to keep costs down - especially important in sectors where deployment has to have a close link to company profitability.
However it is commonplace to have many different versions of images of the same item. As an example, a car manufacturer may have pictures of a vehicle engine used in service, marketing, quality assurance, training and testing applications. As digital asset management becomes recognised as an important control that a company should use, so more attention will need to be spent on reusability and re-purposing of digital assets such as images. JPEG 2000 offers many useful features in this context - proper colour management, compression that can include both lossy and lossless versions of an image in the same file, and extensive options to add user defined and standard metadata to an image file.
In addition, many aspects of scientific and industrial usage involve subsequent processing of a digital image, for example to enhance features or count items. Using any form of lossy compresion for images in this context may create problems - after all the information thrown away during lossy compression is generally that information that is imperceptible to a human eye - not necessarily showing the same characteristics as computer image processing software. It therefore becomes more important to ensure that archival material is stored at the highest fidelity possible - but is still rapidly searchable and viewable during a pre-processing stage for example. Again JPEG 2000 can offer significant advantages in this environment.
Extensive software toolkits are available from a number of vendors which support the new JPEG 2000 standard. These range from the freely available Jasper and JJ2000 software that is linked to Part 5 of the JPEG 2000 standard (Reference Software), to commercial alternatives from KakaduSoft, Aware, Algovision Luratech, Leadtools, Pegasus and others. These allow integration of the comprehensive features of JPEG 2000 into a wide range of products and systems.
Many key applications of the new JPEG 2000 standard will use the Internet, and Internet technologies to distribute images. JPEG 2000 images have a number of properties which make them very suitable for use with the Internet. Typically, Internet users are constrained from downloading large, high quality images because of their physical file size. Often providers of images must create three or more versions of an image, varying from a tiny thumbnail through to a page size image.
Digital cameras have improved in quality and resolution to a level where they are now competing effectively with traditional film. The images they generate are often no longer directly suitable for Internet deployment - the quality and size is wasted on traditional computer monitors. In part, this is because the monitor might show no more than a quarter of the captured image without scrolling, and in part because the colour fidelity of the monitor does not match that of the camera.
Both of these issues are addressed by JPEG 2000 standards. Images saved in JPEG 2000 format can be coded so that the data when transmitted and imaged gradually increases in resolution, starting with a thumbnail, or gradually increases in quality. A combination of these (and other) quality measures can also be achieved - and the user can stop the image transmission once they have enough detail to make their next choice, as the data is ordered in the file in the correct way to simplify its delivery by image servers.
Parts of JPEG 2000 that were created to facilitate these delivery methods are for example:
- Part 8 (JPSEC) deals with image security - for example showing how to use watermarking and other technologies to provide the technical base that many e-commerce applications will need to be able to show their material without risking its piracy
- Part 9 (JPIP) defines new methods to link and deliver the image metadata (information about the image, like its creation time and place) with the image itself, and to deliver under user control the most important pieces of this information first. For example, a doctor looking at an X-Ray could zoom in on areas of interest which could be magnified, or delivered at much enhanced quality long before the rest of the image.
- Part 10 (JP3D) addresses how three-dimensional representations of images could be communicated Part 11 (JPWL) looks at how the particular characteristics of wireless communications and mobile telephony might affect transmission of JPEG 2000 images. It is of course strongly related to the work in JPSEC and JPIP
Traditional surveillance technology has been quite slow to embrace the advantages of digital image processing. In part this has been because the sheer volumes of data have required analogue storage methods such as lapsed time video recorders, and in part because the cost of moving to a digital base have been prohibitive. In the last few years however, costs have fallen dramatically, whilst processing power and capabilities have improved equally fast. This allows many of the shortcomings of traditional surveillance applications to be addressed, whilst also considering many of society's concerns about privacy and intrusion.
Movement detection, and many more sophisticated forms of image analysis can be coupled with new sensor technology to allow much more pro-active monitoring and alarms. Use of 'region of interest' enhancement allow accurate identification of suspects while excluding from analysis, and subsequent public exposure, the innocent bystander. Tight control can be exerted over the user of surveillance technology - for example showing sufficient details to allow recognition of an individual found to have passed a stolen cheque, whilst not permitting enough detail to allow a corrupt viewer of the scene to be able to view and copy a signature being made.
The need for stored evidence to be of a sufficient high quality however also raises concerns and a need for protection against tampering and fabrication. It is very easy within the digital environment to change either subtly or completely aspects of an image, and the metadata surrounding it. Techniques such as encryption and watermarking can be used to help protect against this risk, but there is a real need for well accepted media management techniques which can help reduce risks in this area, for example using trusted third parties and crypto-technology. In addition, it is important that evidence is not segmented, being kept in a single file to avoid the obvious risks of mis-information.
Many of these aspects point to the potential usefulness of JPEG 2000 in this environment:
- the use of Motion JPEG 2000 has obvious advantages in catching sequences of actions, in which the initial view could be at low resolution, switching under the monitor's control to higher resolutions, faster frame rates, and including more metadata and regions of interest
- the file formats defined for JPEG 2000 allow both standardised and user metadata to be stored with the image data
- the new parts of JPEG 2000 extend its usefulness by adding new security support, effective client server communications, and an ability to link its features into an error-prone wireless infrastructure
- as a standard, costs of implementing the technology should be considerably lower than using proprietary technology with less risk of 'lock in'
Document imaging applications are often a trade-off between quality and compression. As technology has improved, and colour become the norm for many publication formats, so user quality expectations have also increased. As there is often a requirement for accurate on-screen viewing, as well as the ability to print high quality facsimiles of an original document, compression requirements are often conflicting. Many documents contain areas which are best communicated in character coded text format (to allow for optimum compression and indexing), together with photographic or half-toned images, graphics and other image types.