International Journal of Modern Research in Engineering & Management (IJMREM) ||Volume|| 1 ||Issue|| 11 || 42-52 || December 2018 || ISSN: 2581-4540
Semi-automatic Picture Book Generation based on Story Model and Agent-based Simulation 1,
Kiyohito Fukuda, 2,Naoki Mori, 3,Keinosuke Matsumoto 1,2,3,
Osaka Prefecture University
-----------------------------------------------------------------ABSTRACT----------------------------------------------------In the fields of artificial intelligence, automatic narrative generation has attracted considerable interest. Lots of studies on narrative generation have been reported such as novel, comic, and picture book. However, most of those reuse original stories, pictures, and sentences. As a result, there is a problem that generated narratives infringe on copyright. In this paper, we focus on the picture book as the narrative because picture book is a mix of images and language. As a first step of automatic story generation without representation mediums, we propose a novel semi-automatic picture book generation method based on story model and agent-based simulation. The computational experiments are carried out to confirm the effectiveness of the proposed method.
KEYWORDS: agent-based simulation, narrative engineering, picture book generation, story model --------------------------------------------------------------------------------------------------------------------------------------Date of Submission: Date, 6 December 2018 Date of Publication: Date 13 December 2018 --------------------------------------------------------------------------------------------------------------------------------------
I. INTRODUCTION Recently, automatic narrative generation by the computer has attracted considerable interest as a challenging problem in the fields of artificial intelligence and natural language processing. Narrative [1] is a creation based on emotions of human and is comprised of a story and a representation medium. When we represent the story by language, narrative is called novels; and it is called comics when we represent the story in images mainly. In this paper, we focus on the picture book which is one of the simplest comics, as narrative. With a view toward realizing effective automatic narrative generation, lots of studies on automatic novel generation based on a case-based reasoning (CBR) [2][3][4] and problem-solving process [5], and automatic comic generation based on picture model to express pictures and images [6][7] have been reported. However, most of those studies have a critical problem that they reuse original stories and sentences directly. As a result, there is a problem that generated narratives infringe on copyright. If we can generate coherent and unexpected narratives without reusing original stories directly, we can solve this problem. Therefore, we have proposed the story model in order to model stories from log data of computational simulation. In this paper, we propose a novel semi-automatic picture book generation method based on story model and agent-based simulation (ABS). Here, ABS is a simulation to investigate actions and interactions between autonomous agents [8][9]. A new picture book is generated by representing a story model which is generated by the simulation log of the ABS.
II. STORY MODEL We generate picture books based on our story model. The most important part of that is called a scene. There are lots of different scenes in the story model. Those differences among scenes are represented as objects that appear in the scene. Each scene has actions and emotions of the object, time, and environment. The story is generated by selecting the next scenes from the all scenes set and transiting scenes from the start scene to the end scene. The ability of modeling branched stories is one of the important features of our story model. The scene has the following informations that are necessary for story generation in hierarchical structure. • The scene such as scene name, place and time.
www.ijmrem.com
IJMREM
Page 42
Semi-automatic Picture Book Generation based…
Figure 1. scene structure of story model • The objects that appear in the scene such as name, age and emotion. • The relations between the objects such as action, identification and causal relation. Fig. 1 shows the scene structure of story model.
III. PROPOSED METHOD In this paper, we propose a semi-automatic picture book generation method based on story model and ABS. The proposed method proceeds as follows. Each step is explained in detail in 3.1 - 3.4. Step 1: Settings of the ABS The environment and parameters of the ABS are set in order to generate log data. Several parameters of the ABS are determined by the user. Step 2: Log data generation via ABS Two types of log data, namely status logs and action logs are generated via the ABS. Step 3: Story model generation Story model is generated from action logs and status logs on the basis of a specific character's log data. Step 4: Picture book generation Picture book is generated based on the story model by using languages and images as representing mediums. Steps 1-3 of the proposed method can be performed automatically. However, step 4 requires human's help in the current system. Thus, the former are referred to as the “computer part” and the latter is referred to as the “user part”. 3.1. Setting of the ABS (Step 1): Fig. 2 shows the flowchart of the ABS algorithm. The ABS algorithm is implemented for the proposed method. Here we represent a real value according to the normal distribution and the uniform distribution of as and . The details of the ABS are explained as follows: Parameter settings There are two types of parameters in the ABS, namely system parameters and user parameters.
www.ijmrem.com
IJMREM
Page 43
Semi-automatic Picture Book Generation based‌
Figure 2. flowchart of the ABS System parameters: , : The field size of the ABS. , : The maximum and minimum value of the friendship value. , : The maximum and minimum value of the familiarity value. , : The maximum and minimum value of emotion. , : The maximum and minimum value of personality. , : The maximum and minimum value of health condition. : Variable coefficient. : Variation range coefficient. : The current number of turns. : The number of character agents. : The number of item agents. : The number of scenes. : Action number. User parameters: : Each agent's name. : Each character agent's age. : Each character agent's sex. Space construction In the ABS, the agent moving space is constructed by a field, which is two-dimensional torus-shaped spaces. All agents and scenes are set on random positions in a certain range of the field. Each of agents and scenes can not be set on the same position. Time management In the ABS, time is managed by turn t and action number . The turn denotes a system parameter that manages the time course in the environment and is updated after all agents performed all actions.
www.ijmrem.com
IJMREM
Page 44
Semi-automatic Picture Book Generation based… The action number denotes a system parameter that manages the order of actions performed in the same turn and is updated each time some agent performs an action. Scenes and agent’s construction First, character agents and item agents are constructed. A character agent is an agent that represents the main character in the stories generated by the proposed method and has the following specific internal states: • attributes: age , sex , personalities related to interaction and movement , emotion , and health condition , where is a natural number, is a binary of man or woman, and are real number of , is a real number of , is a real number of ; • friendship values toward other agent and familiarity values toward other agent where
is a real number of
and
is a real number of
;
• the list of its own item agents. An item agent is an agent that represents the item that is appeared in the stories and has the following specific internal state: • information on character agent that is its owner. Then, both character and item agents have the following internal state: • attributes: names , role in the story , where is “hero”, “enemy”, “ally”, “food”, or “weapon”; • influence value on other agent's movement , where is a real positive value and different for each character agent and item agent; • position in the field . Second, scenes are constructed. A scene is a special position in the field. That is important for the proposed method to generate story model and has the following parameters: role in the story and influence value on other agent's moving , where is “event” or “goal” and is a real positive number. Agent selection An agent is randomly selected from agents that do not perform an action in this turn. If there are character agents in them, agent is randomly selected from them. The selected agent is defined as . Movement An agent always performs an action of movement. It moves to the Neumann neighborhood in the direction of the value according to influence values of other agents and scenes in turn , represented by , Euclidean distance between them, and its own personality. After performed the action of movement, the action number is updated to
.
Interaction If an agent is a character agent and it meets some conditions after movement, it performs an action of interaction with other agent. The algorithm of interaction is given below. 1. If a character agent is set at the same position of other character agents in the field, a character agent is randomly selected form them and is defined as . The friendship value and familiarity value of toward in turn , represented by and , and the emotion of in turn , represented by are updated as follows: (1) (2) (3) (4) (5) Furthermore, if the role of is “hero” and it has an item agent that has the role corresponding to the 's role, its owner is changed from to . In addition, and are updated as follows: (6) (7) 2.
If is set at the same position of item agents that is not belonged to any character agents in the field, an item agent is randomly selected from them and is defined as . 's owner is changed to and the emotion of in turn , represented by is updated as follows:
www.ijmrem.com
IJMREM
Page 45
Semi-automatic Picture Book Generation based…
3.
(8) If the health condition of in turn , represented by is below threshold and have item agents whose role is “food”, an item agent is randomly selected from them and is defined as ℓ. loses ℓ and the health condition of in turn , represented by is updated as follows: (9)
4.
After that, the health condition of ℓ, represented by and position in the field are reset. After performed the action of interaction, the action number is updated to .
Update and end determination of the ABS The friendship values, health conditions, and influence values of all agents and all scenes are updated. Here, A set of character agents is defined as and , , a set of all agents is defined as and , and a set of all agents and all scenes is defined as and . Update equations are shown below: (10) (11) (12) (13) The objective of this update is representing that the importance of character's emotions, items and scenes in the story is changed by the time course. In addition, turn t is updated to t = t + 1. Finally, if a character agent whose role is “hero” is set at the same position of a scene whose role is “goal”, the ABS is halted. If not, it returns to the agent selection. 3.2. Log Data Generation (Step 2): Two types of log data, namely action logs and status logs are generated via the ABS implemented by 3.1. The details of these log types are explained as follows: Action log: Action log information is saved if each agent performs a movement or an interaction in the ABS. The data of action log is {turn, main agent, target agent, position of main agent, action type, action number, the change amount of each parameter caused by the action, item that is exchanged between agents by the action}. Status log: The status logs are saved at the start of each turn and at the end of the last turn in the ABS. These comprise the turn and all parameters of all agents and scenes. 3.3 Story Model Generation (Step 3): The story model is generated from action logs and status logs. The algorithm of story model generation is as follows: 1. A character agent whose role is “hero” is defined as . In addition, a set of scenes whose role is “event” is defined as . The status log of the first turn is defined as . 2. Start scene is generated using . 3. A set of the status log of the last turn when is set at the same position of scene is defined as . 4. The status log of minimum turn in is defined as and this minimum turn is defined as . 5. A set of action log of maximum turn in action log that have action types corresponding to is defined as . 6. The action log of minimum turn in is defined as and this minimum turn is defined as . 7. A scene is generated using , , and the status log of turn .
www.ijmrem.com
IJMREM
Page 46
Semi-automatic Picture Book Generation based… Table 1. Correspondence of scene elements to the log data Scene elements SID Scene name Object Place Time Relation Type OID Name Age Sex Personality Emotion Posture Look Condition Position Action Action number Causal Relation Result Sentiment Identification
Log data No data Scene and action type Agent Scene No data Agent and scene Agent’s type Agent’s Id Agent’s name Agent’s age Agent’s sex Agent’s personality Agent’s emotion Scene Agent’s emotion and health condition Agent’s health condition Scene and agent’s role Action type Action number No data Agent’s parameters Friendship value and familiarity value Agent’s role
8. is updated to . If , it returns to 6. 9. is updated to . If , it returns to 4. 10. The end scene is generated using each agent's status logs of the last turn. Finally, the set of scenes generated in 2, 7 and 10 is defined as the story model. The start scene is only one type. All story models have the same start scene. However, there are four types of end scene, namely simple-end, happy-end, bad-end, and unexpected-end. The end scene is selected from these prepared scenes in accordance with character agent's parameters such as friendship value, familiarity value, and position in the field. Each scene is generated by selecting their elements from choices. The selection of them is in accordance with data of action logs and status logs corresponding to them. However, there are some elements that do not correspond to the log data. Table 1 shows the correspondence of scene elements to the log data. 3.4. Picture Book Generation (Step 4): In this paper, a page of the picture book is generated from a scene of the story model. It is constructed a picture and sentences. The algorithm of picture book generation is as follows: 1. The number of scenes of the story model generated by 3.3 is defined as and the -th scene in the story model is defined as . Here, denotes the start scene and denotes the end scene. Set . 2. Suitable picture parts and sentence templates are selected by referring to ’s elements from the prepared image and sentence template sets. 3. A picture is generated by arranging selected picture parts by human according to . Sentences are generated by combining selected sentence templates and specific elements such as name or emotion of . -th page of picture book is generated by combining the picture and sentences. 4. is updated to , it returns to 2. . If 5. The picture book that is pages is generated by the proposed method.
www.ijmrem.com
IJMREM
Page 47
Semi-automatic Picture Book Generation based… Table 2. Experimental conditions (80, 80) 3 2 4 [-100.0, 100.0] [-100.0, 100.0] [-1.0, 1.0] [0.0, 1.0] [0.0, 1.0] 5 10 Table 3. Correspondence of OID to object attributes OID 1 2 3 4 5
Name Little Red Riding Hood Wolf Hunter meat gun
IV.
Role “Hero” “Enemy” “Ally” “Food” “Weapon”
Sex Woman Man Man null null
Age 10 8 30 null null
COMPUTER EXPERIMENTS
To confirm the effectiveness of the proposed method, we carried out user experiment. In the proposed method, we have to prepare the outline of the generated story in advance. In this experiment, the outline of the story generated by the proposed method was prepared manually referring to “Little Red Riding Hood”. It was defined as follows: Simple-end: The hero arrives her grandmother's house without encountering the enemy. Happy-end: The hero and the ally exterminate the enemy, and they arrive hero's grandmother's house and eat dinner with her. Bad-end: The hero is eaten by the enemy in the way to her grandmother's house. Unexpected-end: The enemy apologizes to the hero and becomes good friend with her, and they arrive her grandmother's house and eat dinner with her. In the proposed method, we implement the GUI system in order for users to set the user parameters. The proposed method can generate over 20000 kinds of picture books. 4.1. Experimental Method: We asked annotators to generate 5 picture books by changing the simulation seed in the proposed method. Table 2 and Table 3 show the experimental conditions and the correspondence of object index called OID to object attributes. Experiments are carried out by 10 annotators. They are university and graduate students. 5 of them are men and the rest are women. After generating picture books, we asked annotators to evaluate them with a 4-point rating of 4 (good) – 1 (bad) from 6 viewpoints: story, unexpectedness, differences from the original story, quality, preference, and variety. We confirmed that annotators knew the story of “Little Red Riding Hood” before this experiment. 4.2. Experimental Results: Fig. 3 shows the example of picture book generated by user experiment. We call this “ ”. Tables 4 - 6 show examples of the story model of . Fig. 3 and Tables 4 - 6 are generated in Japanese and translated into English. Tables 4 - 6 correspond to Fig. 3. For example, the columns whose objects are O-2 or R-2 in table 5 and table 6 correspond to the second page of picture book in Fig. 3. We can understand that the story of Fig. 4 is that Little Red Riding Hood is eaten by the wolf in the way to her grandmother's house. The backgrounds, characters or items are defined by the story model in Fig. 3. The look, personality, emotion and
www.ijmrem.com
IJMREM
Page 48
Semi-automatic Picture Book Generation based… ’s story model (Scene)
Table 4. Example of SID 1 2 3 4 5 6
Scene name Start Acquisition of meat Encounter with Wolf Encounter with Hunter Attack on Wolf Bad-end
Objects O-1 O-2 O-3 O-4 O-5 O-6
Table 5. Example of Objects OID Type Name Age Sex Personality Emotion Posture Look Condition Position
O-1 1 Character Little Red Riding Hood 10 Woman Curious Normal Standing Normal Good Right
R-1 1-1 null null null null null Myself
R-2 1-1 null null null null null Myself
Time Morning Noon Noon Noon Noon Night
Relation R-1 R-2 R-3 R-4 R-5 R-6
’s story model (Object)
O-2 1 Character Little Red Riding Hood 10 Woman Curious Normal Standing Normal Good Left
Table 6. Example of RID OID-OID Action Action number Causal relation Result Sentiment Identification
Place Start In the forest In the forest In the forest In the forest In the forest
O-2 4 Item Meat null null null null null null Good Right
O-3 1 Character Little Red Riding Hood 10 Woman Curious Normal Standing Normal Good Left
…… …… …… …… …… …… …… …… …… …… …… ……
’s story model (Relation) R-2 1-4 Pick up 1 Chain Becomes owner null “Hero”
R-2 4-1 -Pick up 1 Chain -Becomes owner null “Food
R-2 4-4 null null null null null Myself
…… …… …… …… …… …… …… ……
Table 7. Annotators’ evaluation for generated picture books Viewpoints Story Unexpectedness Differences from original story Quality Preference Variety
Mean 3.00 2.98 3.48 2.66 2.50 2.90
Standard deviation 0.96 0.93 0.67 1.01 1.04 0.94
position of the objects are also defined by the story model. Therefore, these show that the proposed method can generate coherent picture books. Next, Table 7 shows the evaluations of annotators. In Table 7, the means of evaluations about story and differences from the original story are high as 3.00 and 2.98. Considering the median of evaluations is 2.50, we can understand that the proposed method can generate different stories from the original story. On the other hand, the mean of evaluations about quality are not high as 2.66 and the standard deviation is over 1.00. In addition, the mean of evaluations about variety is high as 2.90. These results suggest that the proposed method generate stories that are too unexpected to represent as picture book because the ABS depends on its own randomness.
www.ijmrem.com
IJMREM
Page 49
Semi-automatic Picture Book Generation based…
Figure 3. generated picture book
www.ijmrem.com
IJMREM
Page 50
Semi-automatic Picture Book Generation based… V. CONCLUSIONS In this study, we proposed a novel semi-automatic picture book generation method based on story model and ABS as a generation that does not depend on representation mediums. The computational experiments are carried out to confirm the effectiveness of the proposed method. The results of the computational experiments are listed below. • The proposed method can generate appropriate picture books. • The proposed method can generate coherent, difference stories from the original story. In future work, we plan to optimize the settings of the ABS by using evolutionary computation [10] [11] or machine learning [12] [13]. This will allow us to generate picture books based on human preferences and to quantitatively evaluate the story. In addition, current ABS propose a solution where all agents select and perform actions according to their emotions, their personalities and so on. However, these elements have a weak influence while a strong influence is given by random numbers. Therefore, the implementation of a new ABS, where actions, emotions, personalities and roles would have a strong influence for each agent's actions is also an important future work.
VI. ACKNOWLEDGEMENT This work was supported by JSPS KAKENHI Grant, Grant-in-Aid for Scientific Research(C), 26330282.
REFERENCES [1] [2] [3] [4] [5] [6]
[7]
[8] [9] [10] [11] [12] [13]
G. Prince, A dictionary of narratology (University of Nebraska Press, 2003). Turner S. R, The creative process: a computer model of storytelling and creativity (Psychology Press, 1994). Rafael Pérez y Pérez and Mike Sharples, Mexica: a computer model of a cognitive account of creative writing, Journal of Experimental & Theoretical Artificial Intelligence, Vol. 13, No. 2, 2001, 119-139. Pablo Gervás, Belén Díaz-Agudo, Federico Peinado, and Raquel Hervás, Story plot generation based on CBR, Knowledge-Based Systems, Vol. 18, No. 45, 2005, 235-242. J. R. Meehan, The metanovel: writing stories by Computer (Garland Publishing, 1980). Ruck Thawonmas and Tomonori Shuda, Comic layout for automatic comic generation from game log, in Ciancarini, P., Nakatsu, R., Rauterberg, M., Roccetti, M. (Eds.), New frontiers for entertainment computing (Springer US, Boston, MA, 2008.) 105-115. Miki Ueno, Naoki Mori, and Keinosuke Matsumoto, 2-scene, comic creating system based on the distribution of picture state transition, in Sigeru Omatu, Hugues Bersini, Juan M. Corchado, Sara Rodríguez, Paweł Pawlewski, Edgardo Bucciarelli (Eds.), Distributed Computing and Artificial Intelligence, 11th International Conference (Springer International Publishing, Cham, 2014) 459-467. Paul Davidsson, Multi agent based simulation: beyond social simulation, in Scott Moss, Paul Davidsson (Eds.), Multi-Agent-Based Simulation (Springer Berlin Heidelberg, Berlin, Heidelberg, 2001) 97-107. Charles M. Macal and Michael J. North, Tutorial on agent-based modeling and simulation, In Proc. of the 37th Conference on Winter Simulation, WSC '05, 2005, 2-15. B. W. Goldman and W. F. Punch, Parameter-less population pyramid, In Proc. of the 2014 Conference on Genetic and Evolutionary Computation, GECCO'14, (New York, NY, USA, 2014. ACM) 785-792. N. Hansen, The cma evolution strategy: a comparing review, in Lozano, J.A., Larrañaga, P., Inza, I., Bengoetxea, E. (Eds.), Towards a new evolutionary computation (Springer, 2006) 75-102. Richard S. Sutton and Andrew G. Barto, Introduction to reinforcement learning, (MIT Press, Cambridge, MA, USA, 1st edition, 1998). V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, Playing atari with deep reinforcement learning, ArXiv, e-prints, 2013.
www.ijmrem.com
IJMREM
Page 51
Semi-automatic Picture Book Generation based‌ BIOGRAPHIES
Kiyohito Fukuda is a doctor’s course student in the department of Computer Science and Intelligent Systems at Osaka Prefecture University. He received his B.S. and M.S. degrees in engineering from Osaka Prefecture University in 2013 and 2015 respectively. His research interests include natural language processing, and artificial intelligence.
Naoki Mori is an Associate Professor in the Department of Computer Science and Intelligent Systems at Osaka Prefecture University. He received his B.E. degree in 1992 from Department of Physics, Kyoto University, Kyoto, Japan, the M.E. degree in 1994 from Department of Nuclear Engineering, Kyoto University, and the Ph.D. degree in 1999 from the Department of Electrical Engineering, Kyoto University. He was a Visiting Fellow at the University of New South Wales at Australian Defence Force Academy from October 2003 to September 2004. His research interests include evolutionary computation and machine learning.
Keinosuke Matsumoto is an Emeritus Professor at Osaka Prefecture University. He received his B.S. and M.S. degrees in mechanics from Kyoto University in 1976 and 1978 respectively. He worked for Mitsubishi Electric Corporation as a researcher from 1978 to 1996. He received his Ph.D. in electrical engineering from Kyoto University for his work on a knowledge-based approach to power system restoration. His research interests include software engineering, object-oriented technologies, and intelligent systems.
www.ijmrem.com
IJMREM
Page 52