Soz Ucar Yazi Kalir: Social and Collaborative Transcription Platform

Cmpe58A – Cooperation Technologies Project Report

Söz uçar, yazı kalır Çağıl Uluşahin, Dicle Öztürk, Işık Barış Fidaner

07-01-2010

Introduction With the enormous growth of the Web, there has been considerable changes in our lives, to name in this context, our approach to social life and access to information has evolved in almost an uncontrollable way. As our online habits change, the structures on the Web like social gatherings, games, information-bases transform and vice versa. In this respect, how we contribute to and reach this network of networks is of high importance. Not getting lost on the Web where a number of factors inside poses the risk of turning everywhere into an information garbage is highly related to the creation and processing of values, for which it can be said that they are reflected from the real world onto the network-space, transformed and reflected back onto the real world as new values.

Among those values which consist of the content and the actions or the habits of the agents, sharing the content floating around and the media itself constitutes an important place since it helps in both organizing and reforming the Web and gathering people. People from various parts of the world coincide and get connected with ties of some common aim or overlapping interests. These ties not only connect them but also get to be the locomotive of their common productions out of their common interests. Sharing initiated with participation brings new ways to create and produce in cooperative ways with the help of the facilities offered by the Web. However, the type and conditions of this content beside the enthusiasm of the people to share can be a determining factor in what sharing produces around the Web. Image and video sharing on the Web is quite a popular way of value creation, considering their attracting features; however processing and retrieval of this type of content is much harder compared to text within the current borders of technology.

To have a contribution in extending the current limits of technology with the tools of cooperation, a collaborative online transcription system, to be detailed in the coming parts, is developed, which is made up of a network of people who create, shape and enrich the video or speech-based content of the system by firstly contributing to the transcriptions of the videos and secondarily introducing new pieces of them, in addition to evaluating them and also involving in a common channel of communication. The system accommodates a group of people interested in some special domain and is populated with the videos chosen by the very people who are a part of the system. The videos are put into a category upon submission and can also be tagged later. A special editor called subtitle-horse (1) is used to transcribe the videos. The ease of collaboration is facilitated by allowing making small contributions without the requirement of completing the transcription of the whole piece of speech. Those small pieces of contributions accumulate and are combined to be offered as the transcription of the video for the purposes of enhancing the coherence of the content, easy processing and as a side-effect obtaining datasets for speech-to-text and text-to-speech research.

A Scenario The system facilitates videos to: 1. Be listened to and transcribed part by part with the help of a special editor and its timemarkers 2. Have transcriptions which can be integrated from a number of submissions whose alternatives can be evaluated by the members

The members of the system: 1. Provide and describe the content to be transcribed 2. Contribute as transcribers by elaborating on and editing the transcriptions 3. Can tag the videos and documents, contributing to the evolution of the content in a more direct way 4. Report in case of vandalism A visitor is able to: 1. Watch the videos, read their text, and 2. Search the videos based on their transcriptions.

An administrator is going to: 1. Revert harmful changes in transcriptions.

2. Revoke membership if necessary.

A visitor is welcome with the main page (Fig 1) where the categories are listed as links on the left, in the middle is the newsfeed which lists what is going on around the site, and on the right side there is a link to reach the registration form and another one to submit a video, though only members can make submissions.

Figure 1. The initial page is the news feed Starting from the left, the category links are for pages where the videos in that category are listed (Fig 2) with players to watch them, their titles and the names of the submitters.

Figure 2. A category page that lists relevant videos.

Clicking on the title of a video, the page introducing that video opens (Fig 3). There the player, comment submission link, information about the video and the tags that it belongs to are shown.

Beneath them, links to the available transcriptions are located. Each link is for one language and opens the transcription page in that language for that particular video. The 'new language' link beside the transcription links is for initiating a new transcription for some other language. And finally on the video page, we see the comments given for that video so far. Here, the link 'make comment' and transcription links require membership and is directed to the login page if a visitor attempts to open them.

Figure 3. Page of a video.

A visitor can registers to the system using the form below:

Figure 4. Registration form

Successful registration results in the following page:

Figure 5. Registration result page.

For members, right side bar includes links for the videos that that member liked, submitted, transcribed and commented on (Unfortunately, these links are not available in the current state of the system).

The following page shows a capture of video submission which only members can perform.

Figure 6. Video submit page.

And after successful submission, the page for that video opens.

Figure 7. Page of the submitted video

As it can be seen from the transcriptions area on the page above, clicking on one of the given languages or adding a new one initiates the transcription of the video for that language or continues from an already initiated one (in this case, our intention was to show the recorded transcriptions with markers.)

Figure 7. Transcription page opens in a new tab.

The editor subtitle-horse embedded in the system is in the middle of video information and the comments for that transcription. Beneath the player are the dynamic time intervals for subtitles and a text area with some helper editor buttons. The transcriber writes what she hears in the text area and upon hitting create or enter, the text is recorded in the file of that transcription and is appended to the list on the right with its interval.

Completing the whole video is not required so the user may leave the transcription page by clicking on 'save and close' button to record all the fresh text in the database.

(Currently, the application can make use of a static subtitle file which cannot be edited. The idea was to allow later editions of transcriptions and record them as contributions leaving some indicators/markers for intervals).

Implementation S繹z U癟ar Yaz覺 Kal覺r is implemented both in PHP and JSP (Java Server Pages). The PHP implementation is intended only for future use (it can be tested through the following address: http://www.sakareller.net/insan/baris/ders/coco/), and JSP is for the Cooperation-Collaboration course, deployed on ELELE server. In this document, JSP implementation is explained in detail. The code files are organized in several groups: 1- JSP files: These are the files in the project's Web Pages directory. They are divided into two directories, "dynamic" and "forms". They generate the HTML pages that are sent to the client 2- Javascript, CSS, SWF files etc: These are the files that are in folders "css", "js" and "sh" referenced through HTML. They serve several client-side purposes of displaying web pages and providing interfaces for transcription.

The "sh" folder contains subtitle-horse, a transcription tool that is freely available for academic use.

3- Beans: These are special java classes that contain a set of variables throughout a transaction. Three beans are used for form, login and video information.

4- Servlets: These are server-side code that is executed just before generating HTML for the client. They serve various purposes such as database access, checking authentication.

5- Database tables: These are the basic structures that hold all the data used by the system.

6- Java Classes: We used three java classes in package "yazikalir" to encapsulate functions that are specific to various purposes in the system.

Overall view The system has two parties, one is the client side and the other is the application. The client reaches the application from browser and can retrieve or update the content under some conditions like membership.

The application part is made up of JSP files embedded in a structure created with HTML for the interface and partly for database connections; servlets, classes and beans in the middle layer to connect to the database; and the database where all the recordings of the application for both parties are stored. There are additional helper files like Javascript, CSS and SWF files for the interface.

The client is in touch with the JSP parts which are to transmit the requests to the middle layer where the relevant queries for the request or response are decided on and sent to the database. The database is updated through those queries and sends the relevant output to the middle layer. Then, the received output from the database is transferred to the JSP parts to be further sent to the client.

Figure 8. Overall view of the system.

As it can be seen from the below capture of file structure, the JSP files are contained in the folder dynamic under the Web Pages directory where the helper files are kept in their respective places by their types. The middle layer made up of servlets, classes and beans are in Source Packages and lastly the database communication is assisted by Server Resources and some relevant libraries.

Figure 9. Organization of the files in Netbeans project.

JSP files JSP files are server side code that directly generates HTML content to be sent to the client. These files are organized in a way that one JSP file conditionally includes other JSP files to generate the complete HTML file. Which JSP includes whom is shown below:

transcript.jsp

index.jsp includes

includes right_sidebar.jsp

left_sidebar.jsp includes

userbar.jsp

mainmenu.js p video.jsp

success.jsp

vlist.jsp

retry.jsp

vsubmit.jsp newsfeed.jsp

login.jsp faq.jsp

account.jsp

Figure 10. The organization of JSP files Among these JSP files, index.jsp forms the main screen by including three other JSP files, two for sidebars on the left and right, and the third JSP to form the central region. Left sidebar shows the menu, and the right sidebar shows member-specific actions.

The middle JSP is determined by "?page=" parameter, but it is set to newsfeed by default. As the member clicks through the menu items, this parameter is varied to modify the middle JSP to other files such as "?page=vlist&category=lecture", which commands the system to include vlist.jsp in such a way that the member is presented with the list of all videos in lecture category.

1-

account.jsp: This page shows the registration form for membership.

2-

faq.jsp: This page shows general information about the usage of the system.

3- index.jsp: This is the main JSP file, always used except transcription page. Divides the page in two sidebars and one middle area.

4-

left_sidebar.jsp: This is the left sidebar, it directly includes mainmenu.jsp.

5-

login.jsp: This page shows the login form for member-only activities.

6-

logout.jsp: This is an intermediate page, which logs out and redirects to index.jsp.

7-

mainmenu.jsp: This menu shows the hierarchical list of video categories.

8- newsfeed.jsp: This page shows the news feed which lists all past events that took place in the system. It queries the database and generates HTML for each of five event types: submit, comment, contribute, rate, public message.

9-

retry.jsp: This page shows the member registration screen once more if there is an

error.

10- right_sidebar.jsp: This is the right sidebar. It directly includes userbar.jsp.

11- success.jsp: This page is shown to indicate successful registration.

12- transcript.jsp: This is the second main page (other than index.jsp). It is launched in a new tab, and closed after transcription to return to the video page.

13- userbar.jsp: When a member is logged in, this page shows member-specific links which are: Submit a video, Videos I like, Videos I submitted, Videos I transcribed, Videos I commented on.

14- video.jsp: This is a critical page. It shows and provides functions to edit information related to a certain video. It allows to play the video with subtitles, edit existing transcriptions, create a new transcription, tagging and commenting.

15- vlist.jsp: This page shows a list of videos based on different criteria, which include: belonging to a category, having same tag, being liked, transcribed, commented or submitted by the same member.

16- vsubmit.jsp: This page shows a form for submitting new videos. The fields are: Title, description, YouTube/FLV link and an initial set of tags.

17- login.jsp: This page triggers the login handler and redirects to login form or main page depending on the situation.

18- process.jsp: This page processes a registration request, and redirects to DBHandler or retry page in case of success or failure.

19- submit_video.jsp: This page processes a video submission request, and redirects to SubmitVideoHandler or main page in case of success or failure.

Javascript, CSS, SWF files etc These are the files used through generated HTML codes. Among them, javascript files are the active code fragments that are run on the user's browser to provide a little more interactivity.

1. menu.js: Used for the hierarchical menu on the left sidebar 2. swfobject.js: The javascript object that is used to embed SWF into HTML 3. youTubeLoader.js: This is required by the YouTube wrapper SWF CSS files are used by client's browser to determine the style and formatting aspects of displaying HTML: 1. menu.css, hack.css: Used for the menu on left sidebar.

2. style.css: Used to determine overall style of the website SWF files are Adobe Flash programs that are used for several purposes: 1. yt.swf : YouTube wrapper, allows transcription of YouTube videos 2. subtitlehorse.swf: Subtitle Horse, the transcription environment 3. player.swf: JW Player, plays the videos These programs are used to play video files from YouTube and FLV. Subtitle Horse is also a SWF file that is used to provide a user-friendly interface for video transcription.

Beans The beans are special Java classes that are used to keep specific information. The three beans used in the system are: 1. FormBean: This bean is used to keep registration information that is received from the user on registration screen. The variables include first name, last name, username, e-mail and password.

2. LoginBean: This bean keeps the authentication information that is going to be checked against the database, which consists of username and password.

3. VideoBean: This bean is for the video submitting procedure. It keeps variables such as video title, video description, tags and flv/YouTube url.

Servlets Servlets are active code fragments that prepare the environment before forwarding the user to a specific JSP file. These codes usually connect to database either to check some information or insert new entries or update existing information.

1. DBHandler: This servlet runs after registration form is sent. It is responsible to insert a new member into the database.

2. LoginHandler: This servlet is triggered when someone tries to login to the system. It checks the member's credentials to allow or disallow logging in.

3. SubmitVideoHandler: This servlet is used to insert new videos into the system. It receives the information from VideoBean and inserts them into the database.

Database tables There are 14 tables used by the system. Their design is shown on the figure below.

1. publicmessage: This table contains public messages sent by the members. It is related to member table 2. member: This is the table that keeps track of member information. First name, last name, username, email and password as md5.

3. newsfeed: This table holds all events that happened in the system. They are connected to other tables by the combination of event_type and event_ref_id. This id number refers to different tables depending on event type: • event type = 1: video submitted, event_ref_id --> video.vid • event type = 2: video commented, event_ref_id --> comment.commentid • event type = 3: contribution, event_ref_id --> member_transcription.contributionid • event type = 4: transcription rated, event_ref_id --> transcription_rate.rateid • event type = 5: public message sent, event_ref_id --> publicmessage.messageid 4.

video_tag: This table relates three tables, video, member and tag through their id numbers.

5. comment: This table relates members to videos through string content. Comments might also belong to a transcription in a certain language.

6.

video_watch: This table relates videos to the members who watched them.

7.

video_like: This table relates videos to the members who liked them.

8. transcription_rate: This table relates transcriptions to members through given rating values ranging from 1 to 5.

9. member_transcription: This table keeps track of all the contributions made by members on transcriptions of videos in different languages.

10. tag: The list of tags with their string content and id numbers.

11. video: This table keeps the videos in the system. Each video has a submitter member, a language and a category, along with a title, description and url.

12. category: The list of categories that are used to classify videos. The parent category is indicated for some categories (that are subcategories) to create a hierarchical organization.

13. language: The list of languages that are used for transcription in the system.

14. transcription: This table keeps the transcriptions of videos in different languages. Every video can have a single transcription entry for a specific language, whereas there might be several contribution entries for this transcription in member_transcription table.

Java Classes These are a few classes that contain several functions most of which access database for very specific purposes. We are going to examine each of them in detail: 1. global_data: This class holds most general information such as the title and subtitle of the website, the connected database parameters, event types supported by the news feed etc., each in their own constant variables. In addition, it contains the following functions: • getConnection: This is a widely used function. It establishes a database connection and returns it. The caller has to close() this connection after it is done.

• fetchStuff: This is the general function that executes a given query through a given database connection. Several other specific fetch functions call this general function.

• fetchVideoIdsCategory: This function executes a query that returns the list of videos in a certain categories.

• fetchVideoIdsLike: This function executes a query that returns the list of videos that are liked by a certain member of the system.

• fetchVideoIdsTranscribe: This function executes a query that returns the list of videos that are transcribed by a certain member of the system.

• fetchVideoIdsComment: This function executes a query that returns the list of videos that are commented by a certain member of the system.

• fetchVideoIdsSubmit: This function executes a query that returns the list of videos that are submitted by a certain member of the system.

• fetchVideoIdsTag: This function executes a query that returns the list of videos that are tagged with a certain string.

• fetchCategoryString: This function executes a query that returns the name of a given category.

• fetchCategoryIdName: This function executes a query that returns the list of categories along with their names.

• fetchSubcategoryNames: This function executes a query that returns the list of subcategories that have as parent a given category.

• fetchTagsVideo: This function executes a query that returns the list of tags that are connected to a certain video.

• fetchTranscriptionsVideo: This function executes a query that returns the list of existing transcription languages of a certain video.

• fetchTranscriptionsId: This function executes a query that returns the list of existing transcription instances of a certain video • fetchCommentIdsVideo: This function executes a query that returns the list of comments that are written for a certain video.

• fetchCommentIdsLanguage: This function executes a query that returns the list of comments that are written for a certain transcription of a certain video.

• fetchNewsFeed: This function executes a query that returns the list of events that are to be listed on the news feed.

• fetchVideoInfo: This function executes a query that returns all information about a certain video in the system.

• fetchMemberInfo: This function executes a query that returns all information about a certain member in the system.

• fetchContributionInfo: This function executes a query that returns all information about a certain transcription contribution of a user in the system.

• fetchRateInfo: This function executes a query that returns all information about a certain rate of a transcription in the system.

• fetchCommentInfo: This function executes a query that returns all information about a certain comment in the system.

• fetchMessageInfo: This function executes a query that returns all information about a certain public message by a member of the system.

• fetchLanguageInfo: This function executes a query that returns all information about a certain language supported by the system.

• fetchTranscriptionInfo: This function executes a query that returns all information about a certain transcription of a video in a language that is supported by the system.

• generateSubmitEventText: This function executes a query that forms the HTML string that is used to display an event of video submission on the news feed. An example: Ali submitted a video in Documentary category: "Yörünge"

Functions called: fetchVideoInfo, fetchMemberInfo, fetchCategoryString • generateCommentEventText: This function executes a query that forms the HTML string that is used to display an event of commenting on the news feed. An example: Gonca commented on the video "Under The Line": "Nice talk, hard to transcribe though."

Functions called: fetchVideoInfo, fetchMemberInfo, fetchCommentInfo • generateContributionEventText: This function executes a query that forms the HTML string that is used to display an event of transcription contribution on the news feed. An example: Ferit contributed to the Turkish transcription of "Under The Line" Functions called: fetchVideoInfo, fetchMemberInfo, fetchContributionInfo, fetchTranscriptionInfo, fetchLanguageInfo • generateRateEventText: This function executes a query that forms the HTML string that is used to display an event of video submission on the news feed. An example: Eray rated transcription the video "The Database Beneath Consciousness" 3 Star: Moderate Functions called: fetchVideoInfo, fetchMemberInfo, fetchRateInfo, fetchLanguageInfo, fetchTranscriptionInfo • generateMessageEventText: This function executes a query that forms the HTML string that is used to display an event of public message of a member on the news feed.

An example: Ali says: "Please help me transcribe this video!"

Functions called: fetchMemberInfo • putVideoInfo: This function puts in HTML the block of code that displays the video title, the full name of the submitter, and the player to display that video with subtitles.

• generateCategoryMenu: This function calls fetchCategoryIdName and fetchSubcategoryNames to form the HTML code of the menu shown on the left sidebar.

2. user_data: This class contains membership-related functions to access the database: • fetchUserName: This function executes a query that returns the username of a member.

• fetchUserId: This function executes a query that returns the user ID of a member with a certain username.

• userExists: This function executes a query that checks whether a member with a given username already exists.

• authenticate: This function executes a query that checks whether given credentials are correct for authenticating a member.

• insertStuff: This is the general function to execute queries that insert new entries in the database.

• fetchString: This function executes a query that returns a string at the given column in the current row of the resultset.

• fetchInt: This function executes a query that returns an integer at the given column in the current row of the resultset.

• md5hash: This function calculates the MD5 hash of a given string. It is used to keep unreadable passwords in the database.

3. video_data: This is the class that contains video-specific functions such as: • fetchTranscriptionsOfVideo: This function executes a query that returns the list of transcriptions of a certain video.

• fetchLanguages: This function executes a query that returns the list of languages in the system.

• languages: This function executes a query that returns the list of language abbreviations in the system.

Conclusion In this project, an online collaborative transcription system is designed and implemented. The current state of the system does not fully overlap with the initial design due to some technical and partly underestimated difficulties. Some non-major fixes and enhancements will make it a fully working system.

The system is expected to be useful for a wide range purposes, gathering many people. Although transcription is not an easy and funny task, handy tool and useful and quickly-effective output of one's contribution is expected to be a calling factor for participation.

Future Work Technical difficulties encountered on the way are mostly due to the development environment.

We have implemented our design using PHP, a much more convenient language for us, and put it online (2) (for now it does not have its desired domain name). The next task is spreading and populating the site.

Currently, it was not possible to analyse the social aspects of the application since there is not much activation. As it gets populated, we are planning to devise some methods of social network analysis to understand the hidden structures, if any, and to be able to make it more human-friendly by adding some features like recommendation. Before this, social side of the system needs to be worked on. Tools and medium for communication need to be improved and enriched. In addition, combination of small contributions should be handled in a better and more reliable way.

All in all, the system responds our expectations, though at a medium level. Since the idea is appealing for us and the application at hand is flexible to changes, we will be playing with it for a long time.

References (1) (2)

http://www.subtitle-horse.org/ http://www.sakareller.net/insan/baris/ders/coco