p l a t f o r m
Version 1.0 February 26, 2007
S60
S60 Platform: Image and Video Capturing
S60 Platform: Image and Video Capturing | 2
Legal notice Copyright Š 2007 Nokia Corporation. All rights reserved. Nokia and Nokia Connecting People are registered trademarks of Nokia Corporation. Java and all Java-based marks are trademarks or registered trademarks of Sun Microsystems, Inc. Other product and company names mentioned herein may be trademarks or trade names of their respective owners. Disclaimer The information in this document is provided “as is,� with no warranties whatsoever, including any warranty of merchantability, fitness for any particular purpose, or any warranty otherwise arising out of any proposal, specification, or sample. Furthermore, information provided in this document is preliminary, and may be changed substantially prior to final release. This document is provided for informational purposes only. Nokia Corporation disclaims all liability, including liability for infringement of any proprietary rights, relating to implementation of information presented in this document. Nokia Corporation does not warrant or represent that such use will not infringe such rights. Nokia Corporation retains the right to make changes to this specification at any time, without notice. License A license is hereby granted to download and print a copy of this specification for personal use only. No other license to any other intellectual property rights is granted herein.
Version 1.0 | February 26, 2007
S60 Platform: Image and Video Capturing | 3
Contents 1.
Overview .............................................................................................................. 5
2.
Market environment ............................................................................................ 5
3.
High-level and low-level APIs............................................................................. 5
4.
Architecture of S60 multimedia frameworks .................................................... 6
5.
Image capturing use cases ................................................................................ 9
6.
Video capturing use cases ............................................................................... 10
7.
Options for developing applications ............................................................... 10
8.
API availability ................................................................................................... 12
9.
Evaluate this resource ...................................................................................... 13
Version 1.0 | February 26, 2007
S60 Platform: Image and Video Capturing | 4
Change history February 26, 2007
Version 1.0
Initial document release
Version 1.0 | February 26, 2007
S60 Platform: Image and Video Capturing | 5
1.
Overview
This document is a technical overview of the architecture and APIs that support image and video capturing on the S60 platform. Once you have a general understanding of what is possible and how to do it, you can consult the detailed technical documentation available at the Forum Nokia Web site to turn your product and service concepts into reality. With your connected S60 mobile device, you can capture images and videos, edit them, update their metadata, and share them directly with friends or through the Internet. As a software developer, you can take advantage of the open platforms and tools offered by the S60 platform to create new applications and services that leverage existing device features or extend the functionality of S60 devices and platforms.
2.
Market environment
In 2007, more than half of all mobile phones sold in the world will have a camera (Gartner Research), which translates into almost 600 million devices. The vast majority of consumers who have a camera phone use the camera feature at least once in a while, and roughly a third of camera phone owners share their photos by MMS, e-mail, or other means. The major online sharing sites, such as Flickr (http://www.flickr.com/) and YouTube (http://www.youtube.com/), already support uploading of photos and video clips directly from mobile devices; about 10% of camera phone users (based on an IDC study) are already taking advantage of these services. As the quality of both mobile cameras and mobile networks improves, more consumers will be likely to take and share more pictures and video clips with their friends and through the Internet. This opens new opportunities for software developers to create new applications and services to help mobile consumers get more use and enjoyment from their mobile phone cameras.
3.
High-level and low-level APIs
Applications can capture video and images using high-level and low-level APIs. With high-level APIs, an application can capture an image or a video to a file, and then it can use that file for further processing. With high-level APIs, the application does not have control over the parameters used for capturing the image. With low-level APIs, an application can capture an image or a video to memory for further processing. With low-level APIs, the application has control over many parameters used in controlling an image. The S60 platform has one high-level API that allows an application to capture images and videos to files. This API is called CNewFileServiceClient and it is available in the Extensions plug-in for S60 3rd Edition SDK for Symbian OS, for C++, FP1. The API works as follows: When the application requests an image or a video, the device’s camera application is launched in embedded mode to retrieve the picture or video. The user will see the camera application UI on the screen, on top of the client application’s UI. All the normal picture and video settings available in the device’s camera application are available to the user (but the client application cannot change them). After the user captures a picture or video, the control is returned to the client application and the user will see the client application’s UI again. There are two main advantages of using high-level APIs:
Version 1.0 | February 26, 2007
S60 Platform: Image and Video Capturing | 6
1. They provide an easy way for an application to capture images and videos to files. 2. All image and video settings that the device camera application has are available to the user. However, there are two disadvantages as well: 1. Images and videos can only be captured to files; they cannot be captured to memory. 2. The client application has no way of controlling the image and video settings used for capturing. These two advantages and disadvantages are reversed when using low-level APIs for capturing. The main advantages of low-level APIs are: 1. By using low-level APIs, an application has full control over the camera features of a device. 2. Images and videos can be captured to the device’s memory, frame by frame, for further processing (for example, video streaming). Disadvantages when using low-level APIs: 1. With low-level APIs, the application has to implement by itself at least some of the features of the device’s camera application. This includes the onscreen viewfinder and a UI. The application has to capture all the events (such as the user pressing the shutter key) and translate them into function calls. 2. Saving videos to files may be rather complicated because the application must take care of audio-video multiplexing and file encapsulation. The S60 platform also provides a mid-level API, CVideoRecorderUtility, for video capturing. This API allows an application to capture video to a file, but it does not provide a UI.
4.
Architecture of S60 multimedia frameworks
Several APIs are involved in image capturing:
Camera API (CCamera): This API is specified by Symbian and implemented by S60 licensees. The API allows an application to use the embedded camera on a device for taking pictures (and videos). Different image capturing parameters can be set. The image can be captured in JPEG or uncompressed format.
ICL API (Image Conversion Library): This API allows an application to perform various conversions between image formats, such as decoding and encoding an image in different formats (RGB, YUV, JPEG), image rotation, and image scaling.
EXIF API: This API allows an application to read and modify EXIF metadata.
Version 1.0 | February 26, 2007
S60 Platform: Image and Video Capturing | 7
Video capturing uses various image capturing APIs :
Camera API (CCamera): This API is specified by Symbian and implemented by S60 licensees. The API allows an application to use the embedded camera on a device for grabbing camera images, both still images or a stream of images that can be used for the viewfinder, video frames, and games. Different capturing parameters can be set (such as the frame rate and frame size). The frame format can be selected according to the use case: RGB (Symbian bitmaps for viewfinder) or YUV (for video capture).
DevSound API (CMMFDevSound): This API is specified by Symbian. The underlying DevSound framework, containing audio codecs, is partially implemented by Symbian and partially by S60 licensees. The DevSound API is used for capturing encoded audio from the embedded microphone to the memory.
3GP API: This API is S60 internal, and it is not available to developers. The API packs the video and audio streams into an .mp4 or .3gp file format.
MDF DevVideoRecord API (CMMFDevVideoRecord): This API is specified by Symbian. The underlying MDF framework, containing video codecs, is partially implemented by Symbian and partially by S60 licensees. The MDF DevVideoRecord API is used to access HW and SW video codecs. The input is video frames and the output is a compressed video bitstream. In direct capture mode, the DevVideoRecord API gets the frames directly from the camera. Otherwise, the user of the DevVideoRecord API must provide the frames to be encoded. The API supports several codecs, including H.263 and MPEG-4. The MDF DevVideoRecord API has been introduced in S60 3rd Edition. It is, however, not in the S60 SDK but available to developers through partnering through Forum Nokia. This API can also be used for encoding frames that
Version 1.0 | February 26, 2007
S60 Platform: Image and Video Capturing | 8
are not from the camera (for example, a video editor application, or a standby splash screen in video streaming).
MMF Client API (CVideoRecorderUtility): This API is one of the client APIs on top of the Multimedia Framework (MMF). The MMF and the MMF client APIs are specified and implemented by Symbian. The S60 platform implements most of the MMF plug-ins (Controllers) that provide the media recording and playback capabilities of the MMF. The API is available from the S60 SDK. The API takes care of camera video frame capturing, video encoding, audio encoding, A/V synchronization, and file formatting (.mp4, .3gp), which enables applications to capture video and audio to a file from a simple set of commands.
The pictures above show 3 types of components and 3 types of APIs. As indicated by their color, components in the above pictures can be:
Components and APIs in green are implemented (for components) or specified (for APIs) by the S60 platform. These components are present in most of the S60 devices, and they behave in the same or a similar way in all devices.
Components in blue are specified and implemented by Symbian. These components are present in most S60 devices, and they behave in the same or a similar way in all devices.
APIs in blue are specified by Symbian, but the underlying implementation can be done by Symbian, S60, or it can be device specific. APIs that have device specific implementations may behave differently on different devices. Even if just parts of the implementations are device specific, the API can still Version 1.0 | February 26, 2007
S60 Platform: Image and Video Capturing | 9
behave differently on different devices. Such examples are MDF DevVideoRecord and DevSound, as part of their implementation is device specific.
Components in red are device specific. They are implemented in different ways in different devices. APIs on top of these components may behave differently on different devices. Device-specific components are implemented by S60 licensees.
APIs in red are device specific. These APIs are not standardized at all, and they can be different on some devices or be missing from some other devices. Most of such APIs are extensions to existing APIs. Extension APIs often allow access to hardware-specific functions.
APIs in the S60 SDK are normal APIs. Some APIs may only be available from a certain release onwards. All APIs in S60 3rd Edition are guaranteed to be binary compatible with future versions of the S60 platform.
From S60 3rd Edition onwards, some SDK releases also have extension plug-ins. The APIs in these extension plug-ins are not guaranteed to be binary compatible with previous or future releases, and they are not guaranteed to be present in the next release. Some of these APIs are old APIs that will be deprecated in next releases, or they may be early releases of future SDK APIs. Extension plug-ins may also contain APIs that are temporary solutions for accessing different features until a permanent API is released in a future S60 release. APIs in extension plug-ins for S60 SDKs are available only after installing the extension plug-in.
Internal APIs are not available from either the S60 SDK or extension plug-ins. Some of the internal APIs may be partnered through Forum Nokia by Forum Nokia PRO members. These APIs are not guaranteed to be binary compatible with previous or future releases, and they may not be implemented in previous or future releases.
5.
Image capturing use cases
From a software development perspective, the services described below can be thought of as combinations of lower-level use cases, which developers can implement on the S60 platform.
Capture an image to a file: On devices that feature one or more cameras, it is possible to capture a picture to a file.
Capture an image to memory or a file: It is also possible to capture a picture to memory, which can be later saved to a file. It is possible to specify many parameters, such as image format, encoding parameters, zoom, exposure, flash, etc.
Edit image metadata: Once an image has been captured, it is possible to edit its EXIF information. Tags, such as location, can be added.
Transcode an image to a different size, compression level, encoding format: It is possible to modify encoding parameters or the encoding format for an image. This can be used for different purposes, such as creating thumbnails or improving compression.
Version 1.0 | February 26, 2007
S60 Platform: Image and Video Capturing | 10
6.
Video capturing use cases
From a software development perspective, the services described below can be thought of as combinations of lower-level developer use cases, which developers can implement on the S60 platform.
7.
Capture encoded video to a file: On devices that feature one or more cameras, it is possible to capture encoded video and audio to a file. It is possible to specify several parameters, such as the video and audio encoding formats. Metadata can be edited at the same time.
Capture encoded video to memory (for example, for streaming): Capturing video (and audio) to memory is also possible. Capturing raw video and encoding it may need to be performed in two separate steps. The application needs to take care of the audio-video synchronization.
Capture uncompressed video (a stream of camera frames) to memory: It is possible to capture uncompressed video frames to memory for further use (for example, for encoding using a proprietary codec or for motion analysis in games or surveillance applications).
Edit video file metadata: S60 does not currently provide support for this. Applications would need to open the video file, parse the file format, and modify metadata by themselves.
Options for developing applications
The S60 platform supports multimedia development in Symbian C++ and Java™. The platform also supports Flash Lite and Python, with some limited support for video and image capturing. The following tables summarize how to implement the above use cases in Symbian C++, Java, FlashLite, and Python. Use cases (image capturing)
Symbian C++
Java
Capture an image to a file
Use the CCamera API (low level) or the CNewFileServiceClient API (high level).
Use JSR-135.
Capture an image to memory
Use the CCamera API.
Use JSR-135.
Edit image metadata
Use the EXIF API: CExifModify, CExifRead, CExifTag
Use JSR-234 (available in S60 3rd Edition, FP1 and onwards)
Transcode an image to a different size, compression level, encoding format
Use the ICL API: CImageEncoder, CImageDecoder, CImageTransform
Not supported
Version 1.0 | February 26, 2007
S60 Platform: Image and Video Capturing | 11
Use cases (image capturing)
FlashLite 2.1
Python
Capture an image to a file
Not Supported
Import and use the “camera” module.
Capture an image to memory
Not supported.
Import and use the “camera” module.
Edit image metadata
Not supported.
Not supported.
Transcode an image to a different size, compression level, encoding format
Not supported.
Not supported.
Use cases (video capturing)
Symbian C++
Java
Capture encoded video clip to a file
Use CNewFileServiceClient API (high-level), CVideoRecorderUtility API, or CMMFDevVideoRecord API (low-level API, requires partnering from Forum Nokia)
Use JSR-135.
Capture encoded video stream to memory
Use CMMFDevVideoRecord API (requires partnering from Forum Nokia), CMMFDevsound (audio), and CCamera (camera grabbing)
Use JSR-135.
Capture uncompressed video frames to memory
Use CCamera API
Use JSR-135.
Edit video file metadata
Not supported.
Use JSR-234 (available in S60 3rd Edition, FP1 onwards)
Use cases (video capturing)
FlashLite 2.1
Python
Capture encoded video to a file
Launch the device’s camera application to capture a video to a file.
Not supported.
Capture encoded video to memory
Not supported.
Not supported.
Capture uncompressed video to memory
Not supported.
Not supported.
Edit video file metadata
Not supported.
Not supported.
Version 1.0 | February 26, 2007
S60 Platform: Image and Video Capturing | 12
8.
API availability
Availability of C++ APIs: CCamera
CMMFDevVideoRecord
CMMFDevSound
CVideoRecorderUtility
CNewFileServiceClient
CExifModify, CExifRead, CExifTag
CImageEncoder, CImageDecoder, CImageTransform
S60 2nd Edition
Available in the S60 SDK
S60 3rd Edition
Available in the S60 SDK
S60 3rd Edition, FP1
Available in the S60 SDK
S60 2nd Edition
API exists, but it is not used (there are no encoders). A private API with similar purpose can be partnered from Forum Nokia.
S60 3rd Edition
API needs to be partnered from Forum Nokia
S60 3rd Edition, FP1
API needs to be partnered from Forum Nokia
S60 2nd Edition
Available in the S60 SDK
S60 3rd Edition
Available in the extension plug-in for the S60 SDK
S60 3rd Edition, FP1
Available in the extension plug-in for the S60 SDK
S60 2nd Edition
Available in the S60 SDK
S60 3rd Edition
Available in the S60 SDK
S60 3rd Edition, FP1
Available in the S60 SDK
S60 2nd Edition
Not available
S60 3rd Edition
Not available
S60 3rd Edition, FP1
Available in the extension plug-in for the S60 SDK
S60 2nd Edition
Not available
S60 3rd Edition
Available in the S60 SDK
S60 3rd Edition, FP1
Available in the S60 SDK
S60 2nd Edition
Available in the S60 SDK
S60 3rd Edition
Available in the S60 SDK
S60 3rd Edition, FP1
Available in the S60 SDK
Version 1.0 | February 26, 2007
S60 Platform: Image and Video Capturing | 13
Availability of Java APIs: JSR-135
JSR-234
9.
S60 2nd Edition
Yes
S60 3rd Edition
Yes
S60 3rd Edition, FP1
Yes
S60 2nd Edition
No
S60 3rd Edition
No
S60 3rd Edition, FP1
Yes
Evaluate this resource
Please spare a moment to help us improve documentation quality and recognize the resources you find most valuable, by rating this resource.
Version 1.0 | February 26, 2007