|
About
GFX | Q & A
| Get the Spec |
INFO@MINDEGO.COM
A New Innovation in Multimedia
Recent advances in computing and networking technologies
provide the foundation for a new era of innovative communications. Media processing is becoming pervasive
in silicon systems, computer networks, and information management. As a
consequence, consumer’s expectations increase as their appetite
for more sophisticated entertainment and informational experiences
enlarges. However, developing rich interactive experiences is
demanding, and quality content requires the right talent and tools.
Content producers are beginning to look at not only what can be achieved
with media compression and interoperability, but the capabilities of
tools for the development of applications.
Multimedia applications are
compositions of various elements, such as media assets (audio, video,
images, vector graphics), meta data, behavior
logic, and security or encryption mechanisms, that together comprise a
rich, visual interactive experience. The application’s logic is
the element that defines how users experience the content, direct its
flow, or influence an outcome. As content experiences become more
elaborate and complex, the programmatic aspect of multimedia
applications increases in importance. Accordingly, new developments are
emerging that promise to facilitate a new era of amazing multimedia
applications.
GFX – A New Standard
for Multimedia Applications
For
the past two years, Mindego has been a participant and a contributor to
a new extension to the MPEG-4 standard. The standard is officially
known as MPEG-4 Part 21, or ISO/IEC 14496-21, and more commonly
referred to as GFX. GFX, an abbreviation for Graphical Framework
eXtensions, expands on the
work that was begun with MPEG-J to provide for the use of Java classes
within MPEG-4 content. GFX extends the use of Java to enable direct
access to the rendering and composition operations not only on MPEG-4
terminals, but on any multimedia terminal.
GFX is a Java-based framework for interactive multimedia
applications. It is designed purposely for combining audio-video media assets
and 2D/3D vector graphics with programmatically defined logic to
present an audiovisual experience on a screen. It gives application
developers precise control over each instance of the presentation. GFX
is network-agnostic, format-agnostic, and rendering engine agnostic:
• Network
agnostic
– GFX
does not rely on any specific network transport mechanism, but uses the
transport mechanism available in the device (e.g. TCP/IP, UDP, FTP,
HTTP, MPEG-2 TS etc.)
• Media
Decoder agnostic
– GFX
can use the output of any video decoder, independent of the compression
scheme used.
• Application
Format agnostic
– GFX
does not prescribe usage of any application-level file format.
Applications can be made to use file formats such as MPEG-4 BIFS, SVG,
X3D, U3D, Collada, or proprietary formats such as 3DS, SWF, OBJ.
• Rendering
Engine agnostic
– An
application can query which rendering engine is available in the
terminal and use application code targeted for the available or
selected engine. For example, the terminal may support the industry
standard 3D graphics API OpenGL as a low-level API (i.e. close to the
hardware) but at a high-level, the terminal may support more dedicated
application-specific APIs (e.g. game engine renderer).
Leveraging
Java
GFX
uses the Java language to securely
deliver and execute applications on terminals. Java has become near
ubiquitous as a presence in desktop web browsers, has been installed on
over 40 million cellphones worldwide, and is being implemented in PDAs,
set-top boxes, game consoles, and other connected appliances. It is the
technology of choice for the new generation of connected devices that
must perform complex media processing tasks with limited resources.
Numerous standards bodies, including DVB and most recently the Blu-ray
Disc Association, regard Java as an integral component of industry
standards.
The
Promise of GFX
The
goal of GFX is to enable content developers to create advanced
multimedia applications and deploy them efficiently on embedded
devices. It is designed to provide content developers with a
far-reaching flexibility to design interactive applications by
seamlessly combining audio/video content and 2D/3D graphics. But GFX is
also grounded in practicality. The GFX architecture is constructed on
the principle of separating application logic and media assets. This
principle is used extensively in the IT world for Internet
applications, and therefore, is not new. What GFX does it apply the
principle to multimedia and greatly empower the ability to develop rich
applications efficiently. On this basis, GFX holds immense promise as
an enabler for the deployment of bona fide, real-world applications
comprised of rich media contents and interactions.
We believe GFX promises
immense benefits: to content producers, who can design and create their
own applications and leverage them across a wide array of systems;
network operators, who will be able to provision a variety of
applications to different devices in the home; and device makers, who
can build compelling appliances that plug-in to a vast assortment of
existing applications.
From time to time we will
update this section of the Mindego web site to reflect new developments
with GFX and provide more information about the standard and how it is
being used.
GFX Questions and Answers
1. What is GFX and
what makes it unique?
The Graphical
Framework eXtension (GFX) is Part
21 of MPEG-4 International Standard, formally known as ISO/IEC
14496-21. It will become an International Standard at the end of this
year. It enables Java-based applications to access decoders and
graphics resources of a multimedia terminal to compose interactive
presentations to the screen.
GFX is unique in that it does not prescribe any
application description formats or any specific graphic API.
Applications developers are free to develop application-specific formats
adapted to their applications, and to use a terminal's specific graphic
APIs for 2D and 3D graphics.
2. What does it
provide that is not already provided by MPEG-4 systems?
MPEG-4 Systems (Part1) provides timing and
synchronization for elementary streams, and hence, their decoders. In
order to visualize these streams, a compositor is needed to assemble
the presentation, and a renderer is needed to draw the presentation on
a terminal's display. MPEG-4 originally provided BIFS (Binary Format
for Scenes, Part 11) to achieve the composition. By the end of this
year, MPEG will standardize the new LaSeR
(Lightweight Scene Representation, Part 20). While BIFS is based on
VRML/X3D (Virtual Reality Language (ISO/IEC 14772) / eXtensible 3D
(ISO/IEC 19775)), LaSeR is based on W3C SVG
(Scalable Vector Graphics).
BIFS and LaSeR are
application-level formats. They are similar in that they define a streamable file format to describe a scene using a
finite set of features. Being finite, they can only address some applications.
In both cases, applications using those formats can express their logic
using ECMAScript or Java language. However,
these languages can only manipulate features available in their
respective specifications.
Unlike BIFS and LaSeR, GFX
does not provide a format. GFX provides a way to enable applications to
have direct access to the terminal's graphic context and decoders. It
does this using the Java language, which enables complex application
logic to be expressed.
3. Does this provide an important
advantage?
Yes. In a sense, GFX enables applications to replace the
limited compositor prescribed by file formats with a compositor
fine-tuned for specific applications, by the content author. With this
approach, any specific compositor and/or application elements developed
by the content author can be reused for many applications. Furthermore,
they can be used with any application-level format, standard or
proprietary. What GFX is designed to achieve is interoperability at the
application level.
4. Don’t file formats achieve
all of the interoperability I need?
File formats achieve interoperability from a content
point of view, but they do not necessarily achieve consistent
performance on various players. GFX is designed to provide a clean
separation between media assets (audio, video, image, vector graphics,
etc.) and application logic. This separation is a fundamental
breakthrough. It allows logic and media to evolve separately from the
other, and thereby potentially reducing the cost of producing
interactive content dramatically.
5. Why can’t we use MHP to
achieve exactly the same applications?
The Multimedia Home Platform (DVB-MHP) is a standard
developed by the Digital Video Broadcasting (DVB) consortium. MHP is
used in cable/satellite set-top boxes and provides a Java API to access
decoding resources and a 2D graphics.
In terms of composition, MHP provides a very limited set
of features. For instance, graphics layers can only be on top of a
video layer. Moreover, graphics and video layers being independent,
their synchronization is hardly possible.
6. GFX is an MPEG-4 standard. Does
it only work with MPEG-4 media? What about other media formats?
GFX only defines the output representation of visual
buffers for hardware accelerated manipulation (by the application) and
drawing (by the terminal graphic chip). GFX does not define the
compressed formats of visual streams. Therefore, GFX is media agnostic.
7. If it
applies to any media format, why is it an MPEG-4 standard?
GFX development started in MPEG because the
application-level formats in the MPEG-4 specifications were deemed
insufficient to answer the needs of application developers, content
creators, and users. In particular, there was seen to be a need to combine
3D graphics with video, advanced navigation, and special visual effects
on a wide variety of platforms ― from embedded devices to
desktops.
8. Can I use GFX with proprietary
application formats?
Yes, any application-level format as long as you, or a
third party, has written a parser for it.
9. Why does GFX require Java? Which
version of Java is required?
Java has been chosen because it has numerous attributes
that have made it ubiquitous on embedded devices (mobile devices,
set-top boxes, and so on).
Any version of Java can be used.
10. What is the relationship between
GFX and MPEG-J?
GFX uses MPEG-J's
set of APIs to access terminal resources such as decoders and network
resources. The difference between GFX and MPEG-J
resides in the access to rendering resources: MPEG-J
was designed to send and to receive messages from an MPEG-4 compositor
or BIFS. GFX enables direct access to graphic resources. There is no
intermediate layer such as BIFS. In other words, GFX applications
control what they render; whereas with MPEG-J
applications, the MPEG-4 terminal does.
11. What is the licensing structure
for GFX?
GFX, like MPEG-J,
is royalty-free. The GFX specification allows usage of any rendering
API and recommends usage of JSR-239 "Java bindings to OpenGL
ES" and JSR-184 Mobile 3D Graphics (M3G). All are royalty-free
with the exception of M3G, which has a royalty requirement for terminal
developers, but is free application developers.
12. At
what stage is GFX in the standardization process? Where can I obtain a
copy of the specification?
MPEG-4 Part 21 GFX will reach International Standard
status in the last Quarter of 2005.
The specification is public and will be posted on MPEG
public home page at http://www.chiariglione.org/mpeg/working_documents.htm
13. How do
I create applications that are GFX compliant? What authoring tools are
needed?
You need a Java IDE (Integrated Development Environment)
and GFX APIs. Any Java IDE can be used, but this is likely to appeal
only to experienced programmers. For real content development, various
companies are expected to market specific authoring tools designed for
the content production community.
14. The
authoring process seems complex. I am a content creator. Do I now also
need to be a Java programmer?
Yes and No. To explain authoring GFX applications, one
could use the analogy of authoring Web applications. To create a Web
application, a content creator assembles artwork (the web pages),
databases, and logic or control that defines how the artwork will be
updated in response to user events, database changes, and so on. The
content creator may program everything using various languages, or use
the assistance of specific authoring tools with pre-built components
that can be tailored to its needs. We expect GFX to follow a similar
course. At the beginning, the tools are very technical and require
substantial expertise. Over time, authoring tool developers begin to
create the abstract user interfaces that permit content producers to
build increasingly complex applications by dragging and dropping object
into design and workflow models.
15. Why Java? Why can’t I just
use C++?
Java is a mature object-oriented language. It’s
secure, fast, and works on virtually any platform. Downloading C++
applications, like downloading a .exe or .dll file on operating systems such as MS-Windows is
un-secure and prone to malicious code that can disrupt a terminal.
Other languages similar to Java could have been used,
but Java was chosen because of its ubiquitous presence on embedded
devices.
16. In
order to deploy GFX applications, do I need a device that has a 3D
processor? If so, doesn’t this limit the potential of GFX?
GFX is designed to work with hardware accelerated
graphics. It was designed with 3D in mind, in particular, for the needs
of 3D graphics on embedded devices; this is the reason why OpenGL ES (a
lightweight version of OpenGL available on all desktop graphic cards
today) is recommended. Other 2D or 3D graphics APIs can be used,
accelerated by a dedicated processor, or available in software. A
number of fast software implementations exist today on mobile devices
and set-top boxes.
Given the wide availability today of 3D graphics
processing on even low-powered devices, 2D-only rendering is a
limitation of the past. GFX is forward looking at all levels. 3D gives
content creators immensely more powerful rendering and compositing
capabilities. It is intended to enable content creators to infuse
amazing, game-like features into any type of audiovisual experience.
While 3D is slightly more difficult to grasp than 2D, with the
appropriate authoring tools the learning curve can be very fast. In the
end, authoring complexity issues get solved by companies that are
experts in this field. It is our belief that the flexibility gains for
content creators, as well as many other advantages, provide a
compelling basis for the widespread adoption of GFX.
Get the GFX Specification
To obtain
the most recent working draft of the GFX specification, click here,
or point your browser to the following location and navigate to the
section: MPEG-4 Part 21
http://www.chiariglione.org/mpeg/working_documents.htm
|
