MediaTeam - Decade of Research

Decade of Research

1997 – 2007 Ten Years at the Top of the Information Age

Credits Text

Antti Taimisto, Niko Määttä, Timo Ojala, Juhani Toivanen, Jaakko Sauvola

Layout, web design

Antti Taimisto

Supervisors

Marika Leskelä, Timo Ojala

More information, including details on scientific publications, is available in the web version of MediaTeam - Decade of Research at http://www.mediateam.oulu.fi/history/

1 Preface 10 December 2007 Dear Reader, MediaTeam is celebrating ten years of exciting research (1997-2007). Through this publication, you are warmly invited to take a peek at the group’s history and present. During its time, the group has grown to a multi-national and multi-skilled ensemble of highly diversified researchers. As one of the key research groups in the area, MediaTeam is committed to continue solving challenging problems in basic research. For the optimal placement of our research findings, we will continue to work closely with the industry and surrounding communities in the area of applied research. Many of MediaTeam’s findings are utilized by the industry and referenced by the academia today; it has been our intent to openly contribute our research to the immediate setting, the surrounding Northern Finland and the city of Oulu, the companies and communities in Finland, as well as the research community.

I want to express my gratitude and warm thanks to all researchers, partners and sponsors who we have had the pleasure to work with during these joyful years. This has truly been a wonderful journey to new and exciting discoveries, getting to know highly intelligent people and experiencing the priceless feeling of success together!

Sincerely, Dr. Jaakko Sauvola Co-founder of MediaTeam

2 Birth and Infancy

Dr. Jaakko Sauvola, MediaTeam’s Director.

“When your path is blocked, be like water when it encounters an obstacle – flow around it, or, with patience, rise over it” is a phrase Dr. Jaakko Sauvola, MediaTeam’s Director, thinks accurately describes MediaTeam. For him, MediaTeam’s story began in the early 90s, when he became acquainted with the academic world while, at the same time, working in the telecommunications industry. “I began to wonder why there was so little focused research going on in areas that tie together intelligent networking and multimedia processing.”

Sauvola explains. Thus, the idea of MediaTeam started to grow.

Working with his own research in the area and recognizing the needs of the emerging industry, Sauvola decided that it would make sense to start a group to research multimedia in the context of mobility and networking: “It became apparent to me that more research investment was needed – more than one person and a few colleagues can do.” he explains.

MediaTeam was officially established in 1997, after careful planning by Jaakko Sauvola and Timo Ojala. At first, between 1997 and 1999, MediaTeam, or “Media Processing Team” as it was called back then, worked in close cooperation with the Machine Vision Group in the University of Oulu: their research findings were reported together to Infotech Oulu, a local umbrella organization for information technology research. MediaTeam, in its present form, came into being during 1999, when the Machine Vision and Media Processing unit of the Information Processing and Computer Engineering Laboratory in the University of Oulu – 85 persons in total – was divided into two independent research groups, Machine Vision Group and MediaTeam, in order to facilitate efficient research in terms of focus, goals and resources.

From the start, MediaTeam was different from the research groups one could typically find from the academic world at that time. Sauvola wanted to create a team-driven research culture: “Traditionally, researchers did not share their ongoing research with each other. They kept more to themselves, due to academic legacy, the culture of publication and funding.” he explains. “We started with a different process - with a strong identity that we would be a Team with a capital ‘T’ and work with challenges that require much tighter collaboration between researchers than usual. That has allowed us to tackle much bigger problems.” Team work, however, was not the only thing that was different. Since research in universities is mostly funded by taxpayers, Sauvola wanted to make sure that the general public and decisionmakers understood why this research is important. Thus MediaTeam set as its goal not only academic achievements, but also creating new business opportunities, jobs and technology enablers for the emerging industry. From the start, one of the key drivers was to seamlessly mix applied research with basic, academic research, to accelerate research dissemination and, more importantly, to enable researchers to learn faster. In order to make this possible, MediaTeam was created to be a place that was not solely meant for technical people – it would employ people with extremely different backgrounds and skills. An engineering student might work with another, but also with a linguist, mathematician or a graduate in psychology. “This so-called cross-pollination is always useful.” Sauvola says. “We always hired more people than was the plan, but they gave us new ideas, and branched our research to new interesting applications. It is said that it takes ten years to develop into a real research group, but thanks to

our active hiring policy, we exceeded the critical mass and grew into a highly diversified group relatively quickly.” Besides being multidisciplinary, MediaTeam’s research is multicultural, as well: it has employed people from many different parts of the world, and has had regular research visits to foreign partner universities. MediaTeam has also carried its own weight in the field of international cooperation by arranging international events, such as IR 2001 (with the Department of Information Studies), establishing the MUM conference series, hosting it in 2002 and 2007, and hosting the Wireless Cities 2006 conference (with the City of Oulu).

MediaTeam’s research originally took place in five competence teams: computer telephony integration (CTI), content-based multimedia retrieval (CMR), media telephony (MET), mobile information systems (MIS) and multimedia signal processing (MSP). “These teams were born out of necessity - there was a need for this kind of research.” Sauvola explains. “They were based on a vision of structural, networked multimedia. That vision still exists today.” Nowadays there are no more competence teams – currently, research at MediaTeam is focusing on six areas: image and video processing, information hiding, language and speech technology, mobile services, distributed computing and packet networks. When combined, these strategic research areas contribute to solving a vast scale of important problems and enable new solutions to emerge.

For MediaTeam, the role of team spirit cannot be stressed enough – the group’s name contains the word ‘team’ for a reason.

Besides making them share their expertise and ongoing research with each other, Sauvola wanted MediaTeam researchers to have fun together. “We had a totally new approach, since fun is the mother of creativity.” says Timo Ojala, MediaTeam’s Associate Director. “For example, we had these survivor-style trips that I doubt anyone could ever forget.” These trips were not The CMR team leaving for a survivor trip. walks in the park, either – the people were young and continuously at the limits of their knowledge with their research, and all that hard work had to be balanced somehow. Even though ‘team spirit’ is a cliché nowadays, this level of integration of work and personal life was rare back then. Although MediaTeam has grown much since, the ideals behind its name have not been forgotten – it may have become more difficult to organize frequent ‘ad-hoc’ meetings that everybody could fit into their schedules, but the spirit has never been lost.

As for research, MediaTeam is still surfing at the top of the information age, just like it has always claimed. “It’s still valid.” Sauvola says. “’Surfing at the top of the information age’ is all about being able to pick the right things to study – being on top of the next wave.”

3 Video Analysis Researcher Dr. Mika Rautiainen has worked in MediaTeam throughout its existence. During these ten years his career path has progressed quite traditionally from a part-time trainee to a post-doctoral researcher.

Rautiainen started in MediaTeam in the autumn of 1997 as a part-time trainee. In 1998, he visited the NEC Central Research Labs in Tokyo, where he worked on camera-based document analysis systems for six months. After this trip, he continued at MediaTeam as a research trainee, and, in 2000, became a Master’s Thesis researcher. By this time, he had already participated in two projects: Countess, funded by the National Technology Agency, and IDIR. Both projects focused on the research of image retrieval, albeit from a slightly different angle, so it is no wonder that his thesis, titled “Finding semantic knowledge from images in visual information retrieval and surveillance applications”, concerned information retrieval.

Dr. Mika Rautiainen,

MediaTeam Researcher.

In 2001, after receiving his M.Sc. diploma, Rautiainen left for a six month research exchange in the Laboratory for Language and Media Processing in the University of Maryland, located in the USA, near Washington D.C. He worked under the guidance of Dr. David Doermann, the co-director of the laboratory. During his stay there, the topic for his doctoral thesis began to focus on semantic video retrieval, as he was introduced to the TREC video retrieval evaluation at NIST (National Institute for Standards and Technology). His trip to Maryland was a part of the long-running cooperation between MediaTeam and the University of Maryland. The research exchange has resulted in numerous collaborative scientific publications and influenced several doctoral theses.

TREC TREC, the Text Retrieval Conference, aims at annually mapping the development of the research of information retrieval with different retrieval system evaluations. The conference is open to all research groups, and the conference reports can be found from TREC’s web site. The 16th TREC conference was held at NIST (U.S. National Institute of Standards and Technology), Gaithersburg (Maryland), U.S. in 2007.

Originally a conference focused on text document retrieval, TREC broadened its activities towards multimedia in 2001 by including video databases in its evaluations. A considerable number of international research institutes take part in the evaluations. The purpose of their participation in the joint evaluation is to improve the efficiency of content-based retrieval systems in finding relevant information. Thus, the most central aim of the conference is to promote, through science, the development of multimedia retrieval systems on a global scale.

MediaTeam has participated in TREC since 2002.

After returning to Oulu, Rautiainen continued his post-graduate studies and worked as a researcher in MediaTeam.

He continued working towards his thesis, especially in projects funded by the Academy of Finland, namely Semantic Gap and CBIR (Contentbased Information Retrieval). He also managed MediaTeam’s participation in the worldwide TREC video retrieval evaluation.

During his post-graduate time, Rautiainen also participated in other development projects, such as Mobile Kärpät, Oulu Expo, Digital Oulu Cultural Database, Vikings and several subcontracted projects related to mobile devices. In December 2006, Rautiainen defended his

doctoral thesis, which focused on content-based methods that assist users in locating relevant information from heterogeneous video databases. It introduced computational methods for estimating visual and conceptual similarities between video shots and a model for the contentbased browsing of video databases.

Currently, Rautiainen is working at MediaTeam as a post-doctoral researcher, and his plan is to continue that for the next three years and focus on video analysis and its usage in a multimedia information system from the point of view of storage and retrieval. He points out that video analysis can also be utilised in several other application domains, making it a broad and interesting field of research.

The thing that attracted Rautiainen to an academic career was the opportunity to innovate and work with topics that are novel and uncharted. Concretely, research work has meant plenty of reading and investigating various phenomena, as well as writing scientific articles about the findings. In industry-related projects, it has meant tighter control on what should be investigated and how deep a given subject can be delved into. MediaTeam has also given him the opportunity to conduct international research, participate in international conferences and visit foreign research institutions.

Cooperation with Maryland MediaTeam has an agreement on scientific cooperation with the Laboratory for Media and Language Processing (LAMP) at the University of Maryland, USA. LAMP has been a partner in several joint research projects, such as “Cooperative research on computer vision”, “Distributed media processing in hybrid networks”, “Content-based mobile multimedia retrieval”, “Content-based information retrieval” and the CAPNET program.

MediaTeam and LAMP have done joint research on contentbased multimedia retrieval, including joint participation in TREC Video Track in 2001 and 2002. MediaTeam and LAMP also have a research visit program which has facilitated about 30 3–12-month research visits by MediaTeam personnel since 1999. The research visit program has been sponsored by the National Technology Agency.

When asked about MediaTeam’s future, Rautiainen says that he believes that MediaTeam’s focus on creating scientific information will probably produce innovations and information that can be perceived and adopted as new and relevant technological knowledge. He thinks it is quite possible that MediaTeam’s future will bring technological innovations that interest the scientific and international media. The successful establishment of an internationally renowned research brand is not easy, however, and requires decades of research on potentially disruptive technological phenomena. But once established, such reputation presents an opportunity to collaborate with researchers from different cultures while, at the same time, becoming a frontliner in global technological progress. Lastly, Rautiainen encourages beginner researchers to have open-minded dialogue with colleagues, anticipate upcoming tech trends, pursue ideas to their completion and strive towards higher knowledge in their research – that is, to “ask a lot of what’s and why’s.”

Mika Rautiainen wishes the 10-year-old MediaTeam a happy birthday!

Related MediaTeam Projects

CBIR 1/2003 - 12/2006 The objective of the CBIR research project was to reduce the semantic gap by multidisciplinary research based on close international and domestic collaboration between researchers with backgrounds in multimedia signal processing, mathematics, information studies and linguistics, and by integrating information from several media types into an efficient multimedia analysis.

Financiers and Business Partners The Academy of Finland

Semantic Gap 8/2001 - 7/2004 Semantic Gap was a joint project of MediaTeam and the Department of Information Studies in the Faculty of Humanities focusing on the indexing of databases and content-based retrieval of audio and video recordings. Thematically, the project was closely connected with the Vikings project, and its results were extensively tested and applied in the Vikings project.

The central aim of the project was to narrow down the semantic gap between the concept-based and content-based approaches to database indexing. By narrowing the semantic gap, it would be possible to design more and more efficient databases and search engines. The research challenges concerned booming media types, such as digital speech, music, and image, where search criteria often included semantic concepts.

The research questions represented the interface between technology and semantic/cognitive information science, and only a genuinely cross-disciplinary team could hope to tackle the problems. Eventually, the undertaking turned out to be highly successful, as the results were efficiently utilized – the search engine was benchmarked in the international VideoTREC competition, an annual conference series sponsored by the National Institute of Standards and Technology and other U.S. government agencies.

Financiers and Business Partners The Academy of Finland

Vikings 6/2000 - 5/2003 The Vikings project was carried out in cooperation with VTT Electronics. In the project, new contentbased retrieval systems for searches in movie and sound recording databases were developed.

The project’s goals were the development of methods required in content-based multimedia retrieval, the development of novel language technology and the testing of this technology in service applications. Key technologies included digital signal processing, digital image analysis, pattern recognition, visualization, and search engine technology.

The researchers developed new artificial intelligence technologies, by means of which it was possible to detect the emotional state of speakers (with a focus on the Finnish and English languages) from the speech signal almost as automatically and successfully as people do. New image processing techniques were also developed for interpreting video content: changes in color contents in the spatial and temporal domains were measured, and the images were classified accordingly. Finally, the algorithms were integrated into a search engine that combined the audio and video features to achieve higher-level semantic presentations.

Financiers and Business Partners Jutel Nokia OPOY/Finnet Group National Technology Agency

Countess 1/1999 - 12/2000 In the two-year Countess project, researchers developed solutions for content-based image retrieval. The search platform prototype developed by MediaTeam researchers can be used to search for pictures in digital databases on the basis of their content.

Financiers and Business Partners Acta Systems OPOY/Finnet Group National Technology Agency Yritys-Sampo

Selected Publications For more details on the publications below, see the web version of this document at http://www.mediateam.oulu.fi/history/

Rautiainen M, Seppänen T & Ojala T (2006) On the significance of cluster-temporal browsing for generic video retrieval - a statistical analysis.

ACM Multimedia 2006, Santa Barbara, CA, 125-128.

Juuso I & Seppänen T (2006) Novel tools for creating and visualising metadata for digital movie retrieval.

Digital Humanities 2006, Paris, France, 107.

Rautiainen M & Seppänen T (2005) Comparison of visual features and fusion techniques in automatic detection of concepts from news video.

Proc. 2005 IEEE International Conference on Multimedia & Expo, Amsterdam, The Netherlands.

Lilja J, Juuso I, Kortelainen T, Seppänen T & Suominen V (2004) Mitä katsoja kertoo elokuvasta – elokuvan sisäisten elementtien tunnistaminen ja sisällönkuvailu.

Informaatiotutkimus 23(3):59–69 (in Finnish).

Rautiainen M, Ojala T & Seppänen T (2004) Cluster-temporal browsing of large news video databases.

Proc. 2004 IEEE International Conference on Multimedia and Expo, Taipei, Taiwan, 2:751–754.

Rautiainen M, Ojala T & Seppänen T (2003) Cluster-temporal video browsing with semantic filtering.

Proc. Advanced Concepts for Intelligent Vision Systems, Ghent, Belgium, 116-123.

Rautiainen M, Penttilä J, Pietarila P, Noponen K, Hosio M, Koskela T, Mäkelä SM, Peltola J, Liu J, Ojala T & Seppänen T (2003) TRECVID 2003 experiments at MediaTeam Oulu and VTT. Proc. TRECVID Workshop at Text Retrieval Conference TREC 2003, Gaithersburg, MD. Ojala T, Pietikäinen M & Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with Local Binary Patterns.

IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7):971 - 987.

Rautiainen M & Doermann D (2002) Temporal color correlograms for video retrieval.

Proc. 16th International Conference on Pattern Recognition, Quebec, Canada, 1:267 - 270.

Rautiainen M, Penttilä J, Vorobiev D, Noponen K, Väyrynen P, Hosio M, Matinmikko E, Mäkelä SM, Peltola J, Ojala T & Seppänen T (2002) TREC 2002 Video Track experiments at MediaTeam Oulu and VTT. Proc. Text Retrieval Conference TREC 2002 Video Track, Gaithersburg, MD. Ojala T, Kauniskangas H, Keränen H, Matinmikko E, Aittola M, Hagelberg K, Rautiainen M & Häkkinen M (2001) CMRS : Architecture for content-based multimedia retrieval.

Proc. Infotech Oulu International Workshop on Information Retrieval, Oulu, Finland, 179-190.

Doermann D, Sauvola J, Kauniskangas H, Shin C, Pietikäinen M & Rosenfeld A (1997) The development of a general framework for intelligent document image retrieval.

In: Document Analysis Systems II, Series in Machine Perception and Artificial Intelligence, World Scientific, 28 p.

Related Dissertations Rautiainen M (2006) Content-based search and browsing in semantic multimedia retrieval.

Dissertation, Acta Univ Oul C 262, Department of Electrical and Information Engineering, University of Oulu, Finland.

Kauniskangas H (1999) Document image retrieval with improvements in database quality.

Dissertation, Acta Univ Oul C 140, Department of Electrical Engineering, University of Oulu, Finland.

Ojala T (1997) Nonparametric texture analysis using spatial operators, with applications in visual inspection.

Dissertation, Acta Univ Oul C 105, Department of Electrical Engineering, University of Oulu, Finland.

Sauvola J (1997) Document analysis techniques and system components with applications in image retrieval.

Dissertation, Acta Univ Oul C 98, Department of Electrical Engineering, University of Oulu, Finland.

Related Master’s Theses Matinmikko E (2002) Image database browsing system.

M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish).

Keränen H (2001) A mobile retrieval user interface for heterogeneous multimedia document bases.

M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish).

Rautiainen M (2001) Finding semantic knowledge from images in visual information retrieval and surveillance applications.

M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish).

Hagelberg K (2000) Sisältöpohjaisten kuvanhakujärjestelmien hakutekniikat.

Master´s Thesis, Department of Information Processing Science, University of Oulu, Finland (in Finnish).

Koivusaari M (1998) Implementation of content-based document image retrieval system.

M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish).

4

MediaTeam and Convergence Network and service development is one of MediaTeam’s key research fields. The focus of the research has been on convergence – that is, the unification of telephone networks and the Internet. In practice, this is most visible in current mobile phones, which are capable of using both the protocols and resources of the Internet like any other computer.

Overcoming the differences between phones and computers has only been achieved through research that has led to them becoming increasingly alike.

The convergence of mobile terminals and the Internet began with the development of new packet-based network techniques and services and has now reached a point where the mobility of terminals is harnessed and put into use in new applications. Initially, mobile terminals were capable of using Internet services via slow, expensive connections, such as GSM data and GPRS, and only the recent development of Prof. Mika Ylianttila, broadband WLAN and 3G connections has allowed MediaTeam’s Research Manager. their comfortable use. MediaTeam’s research in this area has proceeded logically from studying the early integration of computers and telephones to studying fourth-generation IP phones and applications and developing next-generation networking technology.

The CTI (Computer Telephone Integration) and Princess projects started in early 1998 were MediaTeam’s first looks into this field. The CTI project studied different platform technologies and technical service concepts related to the integration of telephones and computers, while the Princess project focused on mobile multimedia services and media adaptation. The Red Skins project, which started in June 2000, shortly before CTI was concluded, went a step further, focusing on IPv6-based Internet telephony, messaging and multimedia technologies. Later on, in 2004, the Application SuperNetworking (All-IP) project went even further, developing a founding for next-generation networking applications, services and networks.

The concept of a wireless office has also been studied at MediaTeam: “We developed IPbased media phone services in office environments and a software development kit on which services could be built.” says Anja Keskinarkaus, the project manager of the Duchess project of 1999. “Basically, we studied multimedia conference calls in which the participants could, besides talking, see each other, share applications and, say, edit a text file together. We were especially interested in using digital watermarking to incorporate extra information in the data stream – text in speech, for example. This enabled one participant to exchange information with another without the rest of the participants knowing.” Keskinarkaus explains. The Duchess project paved the way for ever more advanced research – a lot was learned of signalling protocols, and the research on information hiding evolved into a separate research branch at MediaTeam. “In the beginning, most of us were very young and inexperienced. The amount we learned during the project was unbelievable, and I’m sure that know-how has spread to the entire research group by now.” Keskinarkaus concludes.

In 2000, after the CTI and Duchess projects, several of MediaTeam’s researchers founded Icecom, a company that sells components for SIP-based communication architectures, server components and terminal software. “One of the most important reasons for founding Icecom was the desire to put the theoretical know-how gathered at the university into practical use.

Other important factors were the enthusiasm of youth and the desire to try something new.

There were also other like-minded people working in MediaTeam who saw that starting our own business would be sensible. And that is how it started.” explains MediaTeam researcher Jani Korhonen, the chairman of the board of Icecom. While the future for Icecom looks challenging, as it competes with international network component manufacturers, its strengths lie in its broad product component assortment and strong software base.

Convergence research comprises of several areas – enabling technologies such as distributed signalling techniques and service platforms, and the testing of end-user-oriented application and service scenarios with prototypes. “The impact of this research will grow more and more as new converged services and Internet service platforms are developed,” says Professor Mika Ylianttila, MediaTeam’s research manager. “Finland’s electronics and IT businesses are outsourcing their production to cheaper countries, so it is essential that Finland’s research and development industry is able to create enough intellectual property in the information services area to encourage new businesses.” Fortunately, MediaTeam is right at the forefront of the research in this field, having received funding and cooperation for several new national and joint European projects from companies such as Nokia, Ericsson and Nethawk, as well as the National Technology Agency and the VTT Technical Research Centre. The quality of MediaTeam’s research on convergence is further underlined by the appointment of Dr. Junzhao Sun, a researcher in several MediaTeam projects related to the field, such as Red Skins, the CAPNET program and All-IP, to the post of Academy of Finland Research Fellow in August 2006. His research concentrates on a cross-layer optimization for sensor networks, aiming to bridge the gap between the technologies needed for modelling, processing and networking by investigating high-level cross-layer optimization techniques in a top-down fashion.

Related MediaTeam Projects

Application SuperNetworking (All-IP) 2/2004 - 2/2007 The Application SuperNetworking (All-IP) project aimed at developing a founding for nextgeneration networking applications, services and networks. The research concentrated on the simultaneous use of several applications in so-called super sessions, mobility and connectivity management, service availability and usability and application interoperability.

The created reference framework enables speeding up application development and the mapping of business opportunities.

Financiers and Business Partners Elektrobit Group IBM National Technology Agency Nokia Serv-It TeliaSonera Finland

Red Skins 6/2000 - 5/2003 The Red Skins project investigated fourth-generation (4G) systems. The research was focused on IP telephony techniques in the 4G environment, mobility management, and measurement of network capacity and data security.

The main aim of the project was to create a novel end-to-end system to produce and test fourth-generation techniques, applications and services. The system comprised of a prototype of an IP telephone network with the required network units and interfaces. Gateway connections to circuit-switched, as well as mobile, telephone and computer networks allowed a variety of value-adding features in a completely new manner. Service provider and user management elements were built on the telephone network, enabling adapted and ubiquitous personal and group services.

Financiers and Business Partners CCC IBM National Technology Agency Nokia Serv-It Sun Microsystems TeliaSonera Finland

Duchess 1/1999 - 3/2002 In the Duchess project, researchers created a wireless office concept, according to which they combined existing technology and traditional office applications with the technology developed in the project into new, innovative services.

One specific research area chosen out of the pilot services was watermarking (information hiding), with many possible applications, such as copy protection and secret messaging. Each type of application set its own requirements for watermarking and information hiding techniques, so researchers in the project studied different possibilities to embed information in audio, images, and video, implementing and testing several different types of algorithms.

Financiers and Business Partners National Technology Agency Nokia TeliaSonera Finland

Princess 3/1998 - 2/2001 In the Princess project, researchers developed a service platform for mobile multimedia services (for example movie, e-commerce, video surveillance and information retrieval), which can be used with terminal devices ranging from next generation mobile phones to multimedia computers. The platform adapts the services according to the user’s terminal device, so service providers do not need to create different versions of the same service for different user groups.

Financiers and Business Partners CCC National Technology Agency Nokia Mobile Phones Nokia Networks TeliaSonera Finland

CTI 1/1998 - 12/2000 In the CTI project, researchers developed software technologies for the integration of telephony and information networks, as well as related terminal devices. The term CTI (Computer Telephony Integration) stands for integration technology which brings the contents of information networks to mobile communications devices and vice versa. As the core of the system, researchers developed a service platform called DAN (Distributed Agent Network), in which users of mobile phones, laptops and desktop computers are served by a personal software agent moving inside the network.

Financiers and Business Partners CCC Koillismaan Yrityspalvelukeskus National Technology Agency Nokia Internet Communications Nokia Networks OPOY / Finnet Group

Selected Publications For more details on the publications below, see the web version of this document at http://www.mediateam.oulu.fi/history/

Sun J (2007) Energy-efficient data gathering in query processing of sensor networks.

Proc. The Second International Conference on Pervasive Computing and Applications, Birmingham, UK, 656-660.

Tienari H, Kaukonen S & Sauvola J (2001) Using mobile code to improve distribution performance in slow wireless environments.

Proc. 9th International Conference on Advanced Computing and Communications, Bhubaneswar, India, 191-198.

Sun J, Riekki J, Jurmu M & Sauvola J (2005) Adaptive connectivity management middleware for heterogeneous wireless networks.

IEEE Wireless Communications 12(6):18-25.

Heikkinen M, Kaukonen S, Palola M, Kostamo N & Strömmer M (2000) A telecom service development platform for heterogeneous communication environments.

Proc. Euromedia 2000 Conference, Antwerp, Belgium, 115-122.

Ylianttila M, Mäkelä J & Pahlavan K (2005) Analysis of handoff in a location-aware vertical multi-access network.

Computer Networks 47(2):185-201.

Koivisto A, Pietikäinen P, Sauvola J & Doermann D (2000) Live multimedia adaptation through wireless hybrid networks.

Proc. International Conference on Multimedia and Expo 2000, New York City, NY, 3:1697-1700.

Howie D, Harjula E, Ala-Kurikka J & Ylianttila M (2005) Harnessing SIP for autonomous mobile peer-to-peer networking.

Proc. 2005 IEEE Global Telecommunications Conference, St. Louis, MO, 2:879-883.

Metso M, Koivisto A & Sauvola J (1999) Content model for mobile adaptation of multimedia information.

Proc. 3rd IEEE Workshop on Multimedia Signal Processing, September 1315, Copenhagen, Denmark, 39-44.

Kassinen O, Harjula E, Pohjanen P, Koskela T, Ala-Kurikka J & Ylianttila M (2005) Group-based content push with dynamic session startup.

Proc. Fourth International Conference on Mobile and Ubiquitous Multimedia, Christchurch, New Zealand, 135-141.

Kia O & Sauvola J (1998) Active multimedia documents for mobile services.

Proc. 2nd IEEE Workshop on Multimedia Signal Processing, December 7-9, Los Angeles, USA: 227-232.

Sun J, Riekki J, Jurmu M & Sauvola J (2005) Adaptive connectivity management middleware for heterogeneous wireless networks.

IEEE Wireless Communications 12(6):18-25.

Ylianttila M, Mäkelä J & Pahlavan K (2005) Analysis of handoff in a location-aware vertical multi-access network.

Computer Networks 47(2):185-201.

Harjula E, Ylianttila M, Ala-Kurikka J, Riekki J & Sauvola J (2004) Plug-and-play application platform: Towards mobile peer-to-peer.

Proc. Third International Conference on Mobile and Ubiquitous Multimedia, College Park, MD, 63–69.

Sun J, Riekki J, Jurmu M & Sauvola J (2004) Design and implementation of Java-based HTTP input channel for integrated WLAN and GPRS networks.

Proc. 12th IEEE International Conference on Networks, Singapore, 258–262.

Howie D (2003) MIPv6 issues in delivering mobile multimedia.

Proc. Advanced Simulation Technologies Conference 2003, Orlando, FL, USA, 49-55.

Sun J & Sauvola J (2003) Mobility management reconsideration: hierarchical model and flow control methodology.

Proc. 14th IEEE International Symposium on Personal, Indoor and Mobile Radio Communications, Beijing, China, 3:2809-2813.

Howie D (2002) Consequences of using MIPv6 to achieve mobile ubiquitous multimedia.

Proc. 1st International Conference on Mobile and Ubiquitous Multimedia, Oulu, Finland, 34-39.

Sun J & Sauvola J (2002) On fundamental concept of mobility for mobile communications.

Proc. 13th IEEE International Symposium on Personal, Indoor and Mobile Radio Communications, Lisbon, Portugal, 2: 799-803.

Metso M, Koivisto A & Sauvola J (2001) Content model for mobile adaptation of multimedia information.

Journal of VLSI Signal Processing 29:115-128.

Sun J, Howie D, Koivisto A & Sauvola J (2001) A hierarchical framework model of mobile security.

Proc. 12th IEEE International Symposium on Personal, Indoor and Mobile Radio Communication, San Diego, CA, 1:56-60.

Related Dissertations Ylianttila M (2005) Vertical handoff and mobility – system architecture and transition analysis.

Dissertation, Acta Univ Oul C 220, Department of Electrical and Information Engineering, University of Oulu, Finland.

Related Master’s Theses Alakurikka J (2007) Context-aware P2P middleware for mobile wellness applications.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland.

Harjula E (2007) Peer-to-Peer SIP in Mobile Middleware Intercommunication.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland.

Kassinen O (2007) Connectivity and session management for networked mobile applications.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland.

Ohtonen J (2007) Group-based navigation application and middleware for distributing context information.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland.

Saloranta H (2003) Architecture for application co-operation during SIP session.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland (in Finnish).

Koskinen E (2003) Controlling media processing in multimedia sessions.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland (in Finnish).

Korhonen J (2002) IP-protocol based telephony network service development by software process.

M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish).

Kostamo N (2002) Superdistribution of digital content in mobile systems.

M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish).

Forstadius J (2002) A model for adaptive multimedia services.

M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish).

LĂśytynoja M (2002) Service platform for adaptive multimedia services.

M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish).

Polvi T (2002) Modelling content for adaptive multimedia services.

M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish).

Tienari H (2002) Modular and intelligent architecture for context-aware telephony services.

M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish).

Koivisto A (2001) Multimedia adaptation for mobile lightweight terminals.

M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish).

Remes L (2001) Adaptive data visualisation in a wireless small terminal device environment.

M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish).

Keskinarkaus A (1998) Prototyping of multimedia transmission system.

M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish).

5

Ubiquitous History

Ubiquitous computing can best be described as the omnipresence of computers: they are integrated into all kinds of everyday objects, such as household appliances and clothing, instead of a single, relatively large device like they are today. These multiple computers are all connected to a network and perform tasks according to the user’s actions, with or without the user being aware of them. Mark Weiser, the “father” of ubiquitous computing, phrased it this way: “The technology required for ubiquitous computing comes in three parts: cheap, low-power computers that include equally convenient displays, a network that ties them all together, and software systems implementing ubiquitous applications”. MediaTeam has contributed particularly to the latter two parts: the network and the software systems.

In terms of software systems, MediaTeam’s contribution has included creating middleware – the software layer Marika Leskelä, between the operating system and its applications MediaTeam’s Program Manager.

– for ubiquitous devices, or, as we call it, “middleware for smart spaces”. Much of this work took place in the CAPNET (Context-Aware PERvasive NETworking) research program. The program comprised of four related projects: Spearhead, Technology Enablers, Challenger and Experiments. Two projects were always in progress simultaneously. The first project pair, CAPNET Spearhead and CAPNET Technology Enablers, was impressive on a national scale – together, the pair formed one of the four national spearhead projects in the NETS (Networks of the Future) program of the National Technology Agency, one of the largest programs in the Agency’s history. “CAPNET was a huge, multidisciplinary program that brought together skilled people from several different faculties, universities and countries. There were more than a dozen researchers from MediaTeam alone.” says Marika Leskelä, the CAPNET program manager.

Jari Forstadius was a project manager in the CAPNET program.

“The research goals were quite broad and wide-ranging.” adds Jari Forstadius, one of the program’s project managers. “We studied things like context-based rules – if you are in a meeting room outside working hours, you probably do not need access to the projector, but this changes if you’re in a meeting – and software that would automatically pick services relevant to your situation from a long list.” Later, the various research themes of the first project pair were focused even further in the second pair, which, though somewhat smaller, created prototypes and demos that contributed a lot to the research of ubiquitous technology. “We created

actual prototype environments which allowed the system to track a person with different kinds of context information and list available services in a specific location – offer a connection to a room’s printer, for example.” Forstadius explains. The impact of the CAPNET program on MediaTeam’s future was considerable, as well: “The CAPNET program gave our postgraduates a glimpse of a ubiquitous world and provided MediaTeam with access to better scientific forums, paving the way for more advanced research on the subject.” Leskelä says. The CAPNET program was also a fine example of international cooperation – it included research visits to the IBM laboratory in Zürich, the Nokia Research Center in Cambridge (USA), the University of Maryland (USA) and the Waseda University in Tokyo, Japan, as well as other cooperation with these two universities.

The research of ubiquitous technology at MediaTeam has, to date, yielded dozens of scientific papers and Master’s Theses on related subjects – service and resource discovery, resource management, middleware architecture, multimedia adaptation and various others. The large amount of quality research, coupled with the fact that many businesses are interested in it, promises an interesting future for this research field.

MediaTeam’s research on ubiquitous computing continues in the UBI program which started in fall 2007.

Related MediaTeam Projects

CAPNET Challenger & Experiments – “CAPNET 2” 1/2005 - 2/2007 In this second CAPNET project, researchers continued working on some of the same things as in the first CAPNET program, but also concentrated on several new research themes. The Challenger project emphasized basic research focuses on familiar themes, such as the utilization of context, service discovery mechanisms and user experience. Context information – time and place – and its use was the most essential idea of the second CAPNET program, but not at the expense of methodological issues and the development of basic technology.

Financiers and Business Partners Capricode IBM Nokia Serv-It TeliaSonera Finland

CAPNET Spearhead & CAPNET Technology Enablers – “CAPNET 1” 1/2002 - 12/2004 The purpose of the CAPNET (Context-aware Pervasive Networking) research program was to create a foundation for new information and communications technologies and businesses in the field. The focus was on context-aware mobile technologies for ubiquitous computing. This kind of information and communications technology and mobile services together formed one of the most promising business fields at that time.

The CAPNET program consisted of four major segments, which were the basis for the development of new services: the processing of context information and the profiling of services, required software technologies and structures, use of content in this new technical environment and new adaptive interface techniques, which allowed the adapting of services according to terminal and environment.

Financiers and Business Partners CCC Hantro IBM Nokia TeliaSonera Finland

Selected Publications

For more details on the publications below, see the web version of this document at http://www.mediateam.oulu.fi/history/

Riekki J, Salminen T & Alakärppä I (2006) Requesting pervasive services by touching RFID tags.

IEEE Pervasive Computing 5(1-2):40-46.

Salminen T, Hosio S & Riekki J (2006) Enhancing Bluetooth connectivity with RFID. 4th Annual IEEE International Conference on Pervasive Computing and Communications, Pisa, Italy, 36-41.

Sun J & Sauvola J (2005) From mobility management to connectivity management.

Proc. 10th IEEE Symposium on Computers and Communications, Cartagena, Spain, 307-312.

Sun J, Riekki J, Jurmu M & Sauvola J (2005) On evaluation of a novel method for adaptive management of heterogeneous wireless networks in mobile client-server communication.

Proc. 16th IEEE International Symposium on Personal Indoor and Mobile Radio Communications, Berlin, Germany, 4:2312-2316.

Forstadius J, Lassila O & Seppänen T (2005) RDF-based model for context-aware reasoning in rich service environment.

Proc. Third IEEE International Conference on Pervasive Computing and Communications Workshops, Hawaii, 15-19.

Khungar S & Riekki J (2005) A context based storage system for mobile computing applications.

ACM Journal of Mobile Computing and Communications Review 9(1):6468.

Salminen T & Riekki J (2005) Lightweight middleware architecture for mobile phones.

Proc. 2005 International Conference on Pervasive Systems and Computing, Las Vegas, NE, 147-153.

Khungar S & Riekki J (2004) A context based data management system for pervasive computing collaborative applications.

Proc. ACM Conference on Computer Supported Cooperative Work, Chicago, IL. Sun J, Riekki J, Jurmu M & Sauvola J (2004) Design and implementation of Java-based HTTP input channel for integrated WLAN and GPRS networks.

Proc. 12th IEEE International Conference on Networks, Singapore, 258262.

Sun J & Sauvola J (2003) Mobility management reconsideration: hierarchical model and flow control methodology.

Proc. 14th IEEE International Symposium on Personal, Indoor and Mobile Radio Communications, Beijing, China, 3:2809 - 2813.

Sun J, Tenhunen J & Sauvola J (2003) CME: a middleware architecture for network-aware adaptive applications.

Proc. 14th IEEE International Symposium on Personal, Indoor and Mobile Radio Communications, Beijing, China, 1:839 - 843.

Doermann D, Karunanidhi A, Parekh N & Rautio V (2003) A platform independent image and video engine.

Proc. 2nd International Conference on Mobile and Ubiquitous Multimedia, Norrköping, Sweden, 35-42.

Mäntyjärvi J & Seppänen T (2003) Adapting applications in mobile terminals using fuzzy context information.

Interacting with Computers 15:512-538.

Ojala T & Ollila M (editors) (2002) Proceedings of the 1st International Conference on Mobile and Ubiquitous Multimedia, Oulu, Finland, ISBN 951-42-6909-8, 112 p.

Mäntyjärvi J & Seppänen T (2002) Adapting applications in mobile terminals using fuzzy context information.

Proc. Fourth International Symposium on Human Computer Interaction with Mobile Devices, Pisa, Italy, 95-107.

Related Master’s Theses Juan Vicente Llácer Olmos (2007) Service and Resource Discovery in Smart Spaces.

M.Sc.thesis, Escola Tècnica Superior de Enginyers de Telecomunicació, Universitat Politècnica de València, Spain.

Jurmu M (2007) Resource management in smart spaces using context-based leases.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland.

Salminen T (2005) Lightweight middleware architecture for mobile phones.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland.

Davidyuk O (2004) Component architexture for context-aware applications.

M.Sc. thesis, Department of Information Technology, Lappeenranta University of Technology, Finland.

6

Waterfall Model of Innovation

The core of research comprises of innovations.

The waterfall model of innovation begins with basic research. The results of this basic research are utilized and combined in applied research, the results of which, in turn, can be further utilized in new commercial products. MediaTeam operates in the first half of this innovation chain, but also takes part in implementing the results with the companies in the field. However, since the results obtained from basic research often cannot be utilized in practical applications as fast as would be necessary, MediaTeam utilizes a faster model in its basic research. This model augments the waterfall model by also concentrating on problems typically encountered in applied research. This unites long-term basic research and faster-paced applied research into a seamless entity. Our research on digital watermarking is a fine example of this.

Prof. Tapio Seppänen, MediaTeam’s Scientific Director.

The hard work of the personnel working on the subject has made MediaTeam one of the key developers in the field of digital watermarking. The broad spectrum of different types of projects has resulted in the accumulation of profound knowledge and the ability to quickly develop state-of-the-art implementations for both national and international business partners.

“Fundamental know-how only develops in longer science projects – projects where there is time to think things through. When we get an R&D project, the understanding is already there. We know what is possible and what is required, and this enables us to quickly reach the end result demanded by the customer,” says Professor Tapio Seppänen, MediaTeam’s Scientific Director and leader of MediaTeam projects related to digital watermarking. “The waterfall model can be seen here: it comes automatically from the range of projects we have, even though I do not like the word itself.” Seppänen concludes.

The research of digital watermarking in MediaTeam began at the end of the 1990’s, when Professor Jaakko Sauvola thought Digital watermarking means hiding information in digital content, such as images and audio, in a way in which the it might be an option for passing additional information is recoverable for a computer, but imperceptible to information in video conferences. Later human senses. This makes digital watermarking a significant on, the emphasis moved to researching part of digital rights management – besides encryption, it is the most important protection method. However, while digital watermarking in the context of encryption must be removed before the product can be used, digital rights management. Internationally, digital watermarks have no such restrictions. Watermarking is the research of digital watermarking had also a new channel for value-added services – for example, a link which can be opened with a mobile phone can be added begun in the mid-nineties, and the first to a sound clip. major publication had appeared in 1996.

Since then, digital watermarking has been researched in MediaTeam in the Duchess, Stardust, Stego, Zirion and Digital Watermarking of Speech and Holograms projects.

Digital Watermarking

Research of digital watermarking has already led to Nedeljko Cvejic’s dissertation on the digital watermarking of audio, Kaisu Mäkelä’s Master’s Thesis on digital watermarking and steganography and Anu Pramila’s Master’s Thesis on watermark synchronization in camera phones and scanning devices. Mikko Löytynoja is currently doing his dissertation on DRM applications and Anja Keskinarkaus on synchronization problems. Adding value to media has also been a field of research in MediaTeam. Anu Pramila has been testing detaching digital watermarks from prints with digital cameras, and Marko Brockman researches embedding watermarks in speech and audio. In practice, the research has led to new algorithms for watermarking images and sound, as well as new application domains.

Digital watermarking is not without its challenges, however, say Anja Keskinarkaus and Mikko Löytynoja. “What makes digital watermarking challenging is the possibility of so-called ‘attacks’ – attempts to remove the watermark, intentionally or unintentionally.” Keskinarkaus says. “Another thing that can be challenging is getting the calculatory complexity to a level where the use of the applications on mobile devices is quick enough.” Löytynoja adds.

Mikko Löytynoja, MediaTeam Researcher.

The actual research consists of three parts: following scientific publications on the subject, developing algorithms and testing them. Testing takes the most time, because processes such as scanning prints are slow, but there simply is no other way to do them. So far, the results include detaching watermarks from both scanned and printed images – with a camera phone, no less – and from a sound recording via speakers. A digital watermark has also been successfully transferred over a phone line.

Cooperation with the industry has included commissions from Nokia and Real Networks. When asked about the overall state of the research of digital watermarks, Löytynoja says: “In the academic world, the research has concentrated mostly on basic research and developing basic algorithms, whereas applications have been developed in the industry. Nevertheless, we have aimed at developing applications, and have already developed applied technology.”

There are already some commercial applications of digital watermarking. There are companies that have Anja Keskinarkaus, been patenting the field quite heavily, and are selling MediaTeam Researcher. licenses to those patents. For example, in the USA, large corporations use digital watermarking to monitor how long their commercials are shown on television. Digital watermarking is also utilized in movie previews: The so-called ‘screeners’ sent to critics, award voters and other movie industry professionals often have a digital watermark that can be used to track down a person who has leaked the screener to the illegal market.

“At the moment, the state of digital watermarking research in MediaTeam is good.” Löytynoja says. The emphasis of the research of digital watermarking in MediaTeam is shifting from digital rights management – even though it is still present in the background – to value-adding services and authentication, where MediaTeam is right at the forefront of research. “A lot of people in MediaTeam are working on the field, and as I see it, the quality of the research is good. More and more publications are appearing, and the field has clearly strengthened from what it was a couple of years ago. There is a lot more knowledge.” Löytynoja concludes.

The research of digital watermarking at MediaTeam continues in the Zirion and Digital Watermarking of Speech and Holograms projects.

Related MediaTeam Projects Digital Watermarking of Speech and Holograms 1/2007 - 12/2010 The project focuses on digital rights management technologies, especially digital watermarking of speech signals and holographic data.

Financiers and Business Partners The Academy of Finland

Zirion 6/2005 - 5/2008 Under the working title ‘Zirion’, this project is a continuation of the research done in MediaTeam on information hiding, albeit with new points of emphasis.

Zirion combines so-called state-of-the-art technologies, such as multi-watermarking, digital rights management and technologies of electronic commerce, in order to create new value-adding services for a mobile environment. MediaTeam researchers have defined five exemplary scenarios, in which the benefits of multi-watermarking are emphasised.

Financiers and Business Partners Capricode Nokia National Technology Agency

Stego 3/2002 - 2/2004 The goal of the project was to develop methods for digital rights management (DRM). The aim of DRM is to efficiently prevent and disclose illegal copying and distribution of digital media products. The targets of illegal copying are more and more often things such as the digital image and music archives on the Internet, digital television broadcasts and DVD products. The research focused specifically on protecting video films and music recordings, because their commercial value is remarkable.

Financiers and Business Partners Nokia Yomi National Technology Agency

Stardust 1/2002 - 3/2005 The Stardust project, conducted jointly by MediaTeam and the Economics unit at the Faculty of Economics and Industrial Management in the University of Oulu, combined knowledge from the fields of technology and economics. The purpose of the project was to research the mechanisms of electrical mobile commerce and the economic models related to the protection of digital contents as parts of future network and application services, as well as to create new service concepts for mobile telecommunications.

Financiers and Business Partners BMG Finland Capricode Embuzz Joulupukki-TV National Technology Agency Nokia Scandinavian A1 Music Targetor

Duchess 1/1999 - 3/2002 In the Duchess project, researchers created a wireless office concept, according to which they combined existing technology and traditional office applications with the technology developed in the project into new, innovative services.

One specific research area chosen out of the pilot services was watermarking (information hiding), with many possible applications, such as copy protection and secret messaging. Each type of application set its own requirements for watermarking and information hiding techniques, so researchers in the project studied different possibilities to embed information in audio, images, and video, implementing and testing several different types of algorithms.

Financiers and Business Partners National Technology Agency Nokia TeliaSonera Finland

Selected Publications For more details on the publications below, see the web version of this document at http://www.mediateam.oulu.fi/history/

Cvejic N, Seppänen T (2007) Introduction to Digital Audio Watermarking.

Cvejic N, Seppänen T (ed.) Digital Audio Watermarking Techniques and Technologies: Applications and Benchmarks, Information Science Reference, Hershey, PA, USA, 1-10.

Löytynoja M, Cvejic N & Seppänen T (2006) Watermark-based counter for restricting digital audio consumption.

International Journal of Signal Processing 3(1):17-23.

Cvejic N & Seppänen T (2005) Digital audio watermarking.

In: Seitz J (ed.) Digital Watermarking for Digital Media, Idea Group, Hershey, PA, 135-181.

Cvejic N & Seppänen T (2005) Reduced distortion bit-modification for LSB audio steganography.

Journal of Universal Computer Science 11(1):56-65.

Cvejic N, Seppänen T & Bull D (2005) Audio watermark detection improvement by using noise modelling.

SEAS Transactions on Systems 4(8):858-864 (invited).

Löytynoja M & Seppänen T (2005) Hash-based counter scheme for digital rights management.

Proc. 2005 IEEE International Conference on Multimedia & Expo, Amsterdam, The Netherlands, 121-124.

Cvejic N & Seppänen T (2004) Spread spectrum audio watermarking using frequency hopping and attack characterization.

Signal Processing 84(1): 207 - 213.

Cvejic N & Seppänen T (2003) Audio watermarking using attack characterization.

IEE Electronics Letters 39(13):1020-1021.

Löytynoja M, Seppänen T & Cvejic N (2003) Experimental DRM architecture using watermarking and PKI. Proc. First International Mobile IPR Workshop: Rights Management of Information Products on the Mobile Internet, Helsinki, Finland, 47-52.

Keskinarkaus A, Cvejic N, Niskanen A, Seppänen T & Sauvola J (2002) Improvements on watermarking images with m-sequences.

Proc. International Workshop on Digital Watermarking, Seoul, Korea, 333-344.

Cvejic N, Keskinarkaus A & Seppänen T (2001) Audio watermarking using m-sequences and temporal masking.

Proc. 7th IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, New York, NY, 227-230.

Seppänen T, Mäkelä K & Keskinarkaus A (2000) Hiding information in color images using small color palettes.

Proc. Third International Workshop on Information Security, Wollongong, Australia, 69-81.

Related Dissertations Cvejic N (2004) Algorithms for audio watermarking and steganography.

Dissertation, Acta Univ Oul C 206, Department of Electrical and Information Engineering, University of Oulu, Finland.

Related Master’s Theses Pramila A (2007) Watermark synchronization in camera phones and scanning devices.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland.

Lähetkangas E (2005) Multiple watermarking of digital images.

M.Sc. thesis, Department of Information Technology, Lappeenranta University of Technology, Finland (in Finnish).

Mäkelä K (2000) Digital watermarking and steganography.

M.Sc. thesis, Department of Electrical Engineering, University of Oulu, Finland (in Finnish).

7

Living Lab and Societal Impact

The concept of “living labs” has gained lots of attention recently. William Mitchell from MIT described it as “a research methodology for sensing, prototyping, validating and refining complex solutions in multiple and evolving real life contexts”. Living labs that foster user-led or user-centric innovations are regarded as key tools in, for example, revitalizing the European innovation system.

In this context, MediaTeam’s most important contribution has been the Rotuaari project.

The general objective of the three-year project was to develop technologies and business models for future context-aware mobile multimedia services and evaluate them empirically with multidisciplinary end user and business-driven living lab research conducted in a real usage environment. The empirical research was conducted in the scope of the service system developed in the project. It comprised of a multi-access network environment, service platforms, and a wide range of prototype services that were used with different kinds of mobile devices.

The system and its services were tested in field trials conducted by real end users in real usage environments. The project carried out seven large-scale field trials and nine smaller concept evaluations, many of them in downtown Oulu. Most of these were a great success, but some failed, as well. The new approach was warmly welcomed by the academia, and the overall scientific contribution of the Rotuaari project was excellent. It produced over 60 international scientific publications, four of which received Best Paper awards in international conferences, as well as over 50 theses, one of which received a national award. The large number of theses underlines the importance of the project’s student workforce. The Rotuaari project also led to the founding of a spin-off company, Brieftec, which commercialized its Mobile Fair Diary application. “The excellent results of the Rotuaari project demonstrate how the pragmatic living lab approach can be successfully combined with scientific goals.”

MediaTeam’s Associate Director, Professor Timo Ojala emphasizes.

A university’s task is threefold: the first, and most important, is research; the second is teaching, Rotuaari project’s field trial office in downtown Oulu. which is based on said research; the third, often overlooked task is societal impact, or “contribution to the society”, as it is often called. During the last ten years, MediaTeam, a relatively small research group in a relatively small university, has set an example of that impact that many others may find hard to follow.

“Our most important societal impact is the panOULU network,” Ojala says. “Today, we have one of the world’s largest open and free-to-use WLAN city networks, used by over ten thousand people every month. It is an exceptional example of how academic research can benefit the whole community.” he muses happily.

Just like in Weiser’s vision for ubiquitous computing infrastructure, the availability of efficient, license-free short-range wireless transmission was deemed mandatory for the implementation of the Rotuaari project. Hence, the building of an outdoor WLAN network in downtown Oulu was started during the preparation of the project in 2002 and continued during the first two years of the project. This work eventually led to the birth of the panOULU network, which was formally founded in October 2003 when the panOULU agreement was signed by the City of Oulu, the University of Oulu, the Oulu Polytechnic and OPOY/Finnet Group. The VTT Technical Research Centre of Finland, Elisa and Netplaza joined the panOULU consortium in 2006. In December 2007, panOULU has some 1000 access points which, in the user’s point of view, appear as one large, uniform network. The open and shared network is also a valuable R&D resource, utilized in a number of pilots and projects. The next challenge is to expand the concept into a regional network that would cover the nearby townships, as well. “PanOULU is a great example of public-private partnership, or the triple-helix model as it’s called in the EU, where the public sector, educational institutes and the private sector work together for the common good.” Ojala concludes.

MediaTeam has also influenced the world around it on a smaller scale by building the Mobile Forum Oulu Showroom in Science Centre Tietomaa in Oulu as a part of the MFO Showroom project of 2000. One of the key attractions of the showroom was a large collection of mobile phones from early models – which resembled more bricks than phones – to then-cutting-edge technological wonders. The exhibition was also digitized into the Internet, and the usability of different models was studied extensively. If nothing else, the project taught the researchers that building a showroom that would stand the test of time with limited resources was no easy task. A similar contribution was made in the Oulu Expo project, where MediaTeam developed a future video browser for the Oulu Expo exhibition displayed at Science Centre Tietomaa.

Living lab research and the creation of new, shared ubiquitous computing resources for the whole community will continue in the UBI program which started in fall 2007.

Related MediaTeam Projects

panOULU 10/2005 - 12/2007 The goal of the panOULU project, co-financed by the University of Oulu and the City of Oulu, was to offer the expertise and knowledge of the Department of Electrical and Information Engineering of the University of Oulu for use in the COMPETENCE Oulu 400 program and the panOULU citizens’ network. This meant, for example, piloting centralized remote administration solutions, wireless point-to-multipoint links and mesh networks in the implementation of the panOULU network.

In collaboration with the COMPETENCE Oulu 400 program, the Department of Electrical and Information Engineering in the University of Oulu participated in maintaining the network base stations, developing the network administration tools and procedures, following up and analyzing network usage, sharing knowledge about network administration and maintenance between various bodies operating within the network, and surveying alternative maintenance expenses for future decision-making.

Financiers and Business Partners City of Oulu

Rotuaari 6/2003 - 5/2006 The Rotuaari project aimed at developing and testing technologies and earnings models for context-aware mobile multimedia services of the future. The project consisted of three segments: the development and implementation of a service platform, field testing with actual end users and a value chain, in which the technology providers contributed to the process with their own view of what could be done with current and future technologies. Respectively, shopkeepers, public bodies, and users all provided their own opinion on what was relevant to their activity, business or otherwise.

The services and technologies were tested in a ground-breaking real end user environment, now called Living Lab, in which the people of Oulu, tourists and visitors all acted as test users. The field tests gave valuable information on the technical performance of the service platform, the usability of the services, actual consumer needs, consumer behaviour, and the functioning of the earnings models. The service architecture and services were developed further based on the results from the field tests.

Financiers and Business Partners City of Oulu Healthpoint Technologies Kaleva Leiki National Technology Agency Nokia OPOY / Finnet Group Oulun Liikekeskus ry Solid Information Technology Targetor TeliaSonera Finland

MFO Showroom 4/2000 - 3/2003 In the MFO Showroom project, researchers created both a virtual and an actual Mobile Forum Showroom, both of which presented the evolution of mobile phones starting from ages-old bricks and ending with then-hot technology. The project also included usability research on these mobile phones.

Financiers and Business Partners CCC Nokia National Technology Agency TeliaSonera Finland Viestimaa

Selected Publications For more details on the publications below, see the web version of this document at http://www.mediateam.oulu.fi/history/

Korhonen J, Ojala T, Ristola A, Kesti M, Kilpelänaho V, Koskinen M & Viippola E (2007) Mobile Fair Diary - Hybrid interface for taking, browsing and sharing context-aware notes.

Personal and Ubiquitous Computing 11(7):577-589.

Related Master’s Theses

Kukka H & Ojala T (2006) mobileDOK: Culture in your pocket.

Proc. 3rd IEE Mobility Conference 2006, Bangkok, Thailand.

Parhi P (2006) Two case studies in the design and evaluation of a mobile interactive system.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland. SIGCHI Finland’s Thesis of The Year Award in 2007.

Ojala T, Hakanen T, Mäkinen T & Rivinoja V (2005) Usage analysis of a large public wireless LAN. Proc. 2005 International Conference on Wireless Networks, Communications and Mobile Computing, Maui, HI, 1:661-667.

Costa J (2006) Mobile Lecture Interaction - Making education and technology click.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland.

Scheible J & Ojala T (2005) MobiLenin - Combining a multi-track music video, personal mobile phones and a public display into multi-user interactive entertainment.

Proc. ACM Multimedia 2005, Singapore, 199-208. Best Arts Paper Award.

Kukka H (2006) Digital Oulu Cultural Database - A study on digital cultural heritage archives.

M.Sc. thesis, Department Information Processing Science, University of Oulu, Finland.

Sutinen T & Ojala T (2005) Case study in assessing subjective QoS of a mobile multimedia web service in a real multi-access network.

Proc. Thirteenth International Workshop on Quality of Service, Passau, Germany, 298-312.

Lukkari J (2004) A usability analysis of a remote payment service for context-aware mobile environment.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland (in Finnish).

Aalto L, Göthlin N, Korhonen J & Ojala T (2004) Bluetooth and WAP Push based location-aware mobile advertising system.

Proc. Second International Conference on Mobile Systems, Applications and Services, Boston, MA, 49–58.

Sutinen T (2004) End user service quality in multi-access networks.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland.

Aittola M, Parhi P, Vieruaho M & Ojala T (2004) Comparison of mobile and fixed use of SmartLibrary.

Proc. 6th International Conference on Human Computer Interaction with Mobile Devices and Services, Glasgow, Scotland, 383–387.

Ojala T, Korhonen J, Sutinen T, Parhi P & Aalto L (2004) Mobile Kärpät – A case study in wireless personal area networking.

Proc. Third International Conference on Mobile and Ubiquitous Multimedia, College Park, MD, 149–156.

Aittola M, Ryhänen T & Ojala T (2003) SmartLibrary - Location-aware mobile library service.

Proc. Fifth International Symposium on Human Computer Interaction with Mobile Devices and Services, Udine, Italy, 411-416.

Ojala T, Korhonen J, Aittola M, Ollila M, Koivumäki T, Tähtinen J & Karjaluoto H (2003) SmartRotuaari - Context-aware mobile multimedia services.

Proc. 2nd International Conference on Mobile and Ubiquitous Multimedia, Norrköping, Sweden, 9-18.

Peltonen J, Ollila M & Ojala T (2003) TimeMachine Oulu - Dynamic creation of cultural-spatio-temporal models as a mobile service.

Proc. Fifth International Symposium on Human Computer Interaction with Mobile Devices and Services, Udine, Italy, 342-346.

Aittola M (2003) Location-aware services in mobile devices.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland (in Finnish).

8

Humanism and Technology

Although current statistical methods used in different areas of language technology may already solve many problems in everyday life, computers do not really understand natural language yet. The mere possibility of using natural spoken or written language in the interaction between humans and computers noticeably increases the usability of different applications.

Automatic speech and text recognition are key technologies in this kind of interfaces: in the future, computers could, for example, recognize the emotions and attitudes of their users directly from the acoustic structure of speech.

Thematically and scientifically, the establishment of MediaTeam’s Language and Audio Technology team in 1999 was a response to the urgent need for an integrated approach to the research of digital databases of text and Dr. Juhani Toivanen, audio material. From the very beginning, the aim of the MediaTeam Researcher. team’s research was to utilize and develop methods for pattern recognition, digital signal processing and the automatic interpretation, recognition and processing of speech and audio signals and text within the framework of artificial intelligence.

The most important application areas still include new search engines for movie and sound recordings databases, as well as computer tools for the design and description of audiovisual content. The results of MediaTeam’s research in this field have not gone unnoticed: Dr. Juhani Toivanen, a member of the team, was appointed to the post of Academy of Finland Research Fellow for August 2004 - July 2009. In his research, Toivanen studies the vocal correlates of emotions in spoken Finnish and their multi-parametric prosodic analysis, broadening and deepening his earlier research on phonetic and linguistic correlates in emotions conveyed by Finnish speech.

The Language and Audio Technology team was established in its initial form when Professor Tapio Seppänen, who had defended his doctoral dissertation at the University of Oulu some ten years earlier, left VTT Electronics to rejoin the ranks of the Faculty of Technology in the University of Oulu. At VTT Electronics Professor Seppänen had worked as a senior and chief research scientist in 1993-1998 and as the head of the Electronic Circuits and Systems division in 1998-1999.

The mission of the Language and Audio Technology team was essentially cross-disciplinary; it was therefore necessary to recruit a group of promising researchers representing different areas and interests. In 1999, Ilkka Juuso, Kai Noponen and Eero Väyrynen joined the team as young students of electrical engineering. Pertti Väyrynen, who held an M.A. degree in English Philology at the time, also joined in. Väyrynen had a background in linguistics and considerable experience in language technology: his company, PhoneSoft, produced translation software systems. Dr. Juhani Toivanen made his first contribution in 2000 as a part-time researcher; he had a background in English Philology and language teaching, and had also worked as a university lecturer in phonetics and general linguistics. The funding of the team’s research was – and still is – almost entirely dependent on outside sources, such as the National Technology

Agency and the Academy of Finland, significantly backed up by funding from industry.

Currently, the Language and Audio Technology team consists of one professor, five fulltime researchers and a part-time researcher. It participates actively in the Osaava PohjoisSuomi (Capable Northern Finland) project, a large undertaking of northern universities and polytechnics which aims to promote cross-disciplinary research in Northern Finland within key areas such as wellness technology, ICT, and travel industry. MediaTeam’s contribution – the development of new algorithms and software for digital signal analysis and classification – is significant: a lexicon and phraseology system comprising of several languages and genres relating to health issues and enabling an automatic translation via wireless devices would be a great asset in a clinical context.

Related MediaTeam Projects

Prosody of Emotions 1/2003 - 12/2006 The Prosody of Emotions project was a part of a more extensive program called “Multidisciplinary research project on the expression of emotion in spoken Finnish” that consisted of three partner departments and laboratories from the Helsinki University of Technology, the University of Oulu and the University of Tampere.

The project aimed to deepen and broaden the linguistic research of emotional correlates in spoken language and find potential targets of application, especially in the field of information technology. A specific aim was to investigate the phonetic and phonological correlates that convey emotions in spoken Finnish and to develop pattern recognition methods for classification of emotional speech signals. The top-level goal of this research was to develop the first systematic descriptive model for spoken Finnish that would take several linguistics layers into account: the aim was to create a theoretical model which could be utilized in multi-dimensional speech corpuses and search engines. In the project, speech signals were analyzed both instrumentally (with speech algorithms) and auditorily to investigate their phonological structure. The results of the project basically determined, for the first time, to which degree the automatic classification of emotion from spoken Finnish is attainable.

In connection with this project, the MediaTeam Emotional Speech Corpus was collected. The database is the largest emotion speech corpora for spoken Finnish, and also one of the largest globally. Also in connection with the project, the f0Tool – a speech analysis software utilizing MATLAB – was developed. The f0Tool has been verified to be a highly reliable software package for automatic prosodic analysis of large quantities of speech data; it can also be used on speech data produced in very demanding conditions.

Financiers and Business Partners The Academy of Finland

Semantic Gap 8/2001 - 7/2004 Semantic Gap was a joint project of MediaTeam and the Department of Information Studies in the Faculty of Humanities focusing on the indexing of databases and content-based retrieval of audio and video recordings. Thematically, the project was closely connected with the Vikings project, and its results were extensively tested and applied in the Vikings project.

The central aim of the project was to narrow down the semantic gap between the concept-based and content-based approaches to database indexing. By narrowing the semantic gap, it would be possible to design more and more efficient databases and search engines. The research challenges concerned booming media types, such as digital speech, music, and image, where search criteria often included semantic concepts.

The research questions represented the interface between technology and semantic/cognitive information science, and only a genuinely cross-disciplinary team could hope to tackle the problems. Eventually, the undertaking turned out to be highly successful, as the results were efficiently utilized – the search engine was benchmarked in the international VideoTREC competition, an annual conference series sponsored by the National Institute of Standards and Technology and other U.S. government agencies.

Financiers and Business Partners The Academy of Finland

Vikings 6/2000 - 5/2003 The Vikings project was carried out in cooperation with VTT Electronics. In the project, new contentbased retrieval systems for searches in movie and sound recording databases were developed.

The project’s goals were the development of methods required in content-based multimedia retrieval, the development of novel language technology and the testing of this technology in service applications. Key technologies included digital signal processing, digital image analysis, pattern recognition, visualization, and search engine technology.

The researchers developed new artificial intelligence technologies, by means of which it was possible to detect the emotional state of speakers (with a focus on the Finnish and English languages) from the speech signal almost as automatically and successfully as people do. New image processing techniques were also developed for interpreting video content: changes in color contents in the spatial and temporal domains were measured, and the images were classified accordingly. Finally, the algorithms were integrated into a search engine that combined the audio and video features to achieve higher-level semantic presentations.

Financiers and Business Partners Jutel Nokia OPOY/Finnet Group National Technology Agency

Selected Publications

For more details on the publications below, see the web version of this document at http://www.mediateam.oulu.fi/history/

Toivanen J, Waaramaa T, Alku P, Laukkanen A-M, Seppänen T, Väyrynen E, Airas M (2006) Emotions in /a:/: a perceptual and acoustic study.

Logopedics Phoniatrics Vocology, 31: 43-48.

Toivanen J & Waaramaa T (2005) Tone choice and voice quality of dispreferred turns in the English of Finns.

Logopedics Phoniatrics Vocology, 30: 181-184.

Toivanen J, Seppänen T, Väyrynen E (2004) Automatic discrimination of emotion from spoken Finnish.

Language and Speech, 47: 383-412.

Iivonen A, Seppänen T, Noponen K & Toivanen J (2004) Puhujan temporaalisen äänialan visualisointisovellus.

Puhe ja kieli 24(1):5–16 (in Finnish).

Toivanen J (2004) Pitch dynamism of English produced by proficient non-native speakers: preliminary results of a corpus-based analysis of second language speech.

Proc. FONETIK 2004, Stockholm, Sweden, 48–51.

Seppänen T, Toivanen J & Väyrynen E (2003) MediaTeam Speech Corpus: a first large Finnish emotional speech database.

Proc. 15th International Congress of Phonetic Sciences, Barcelona, Spain, 3:2469-2472.

Suomi K, Toivanen J, Ylitalo R (2003) Durational and tonal correlates of accent in Finnish.

Journal of Phonetics, 31: 113-138.

Toivanen J & Seppänen T (2002) Prosody-based search features in information retrieval.

Proc. Fonetik 2002, Stockholm, Sweden, 105-108.

Väyrynen P, Seppänen T, Noponen K & Juuso I (2002) On the usefulness of linguistic knowledge in different areas of application in language technology.

Informaatiotutkimus 3/2002: 59-66 (in Finnish).

Väyrynen P, Peltola J & Seppänen T (2000) Enhancing phoneme recogniser performance with a simple rulebased language model.

Proc. STeP 2000 - Finnish Artificial Intelligence Days, Espoo, Finland, 171-178.

Related Dissertations Väyrynen P (2005) Perspectives on the utility of linguistic knowledge in English word prediction.

Dissertation, Acta Univ Oul Humaniora B 67, Faculty of Humanities, University of Oulu, Finland.

Related Master’s Theses Väyrynen E (2005) Automatic emotion recognition from speech.

M.Sc. thesis, Department of Electrical and Information Engineering, University of Oulu, Finland.