CEDEC 2019 Speakers Interview Series The 3rd installment:Our talents go beyond creating the powerful!Creating Enemy AI for a 3D Action Game Using Deep Learning

Facebook Share Twiiter Share Linkedin Share Reddit Share Tumblr Share

At CEDEC2019 held in September, a group of developers from Luminous Productions presented a variety of topics pertaining to the latest game technology. We will be releasing a series of speaker interviews on an irregular basis to reveal additional details that the speakers did not have the chance to discuss during their presentations, along with the speakers' thoughts and sentiments behind the development.


Tatsuhiro Joudan, Programmer

An era where AI creates AI

―Please tell us what made you decide to give your presentation on this particular theme?

Joudan:We struggled quite a bit with creating AI during the development of FINAL FANTASY XV, so my initial thought was "perhaps we can create AI with lesser costs by using deep learning, since everyone seems to be talking about it these days." With that, I set out to do my research in the hope of pushing our Luminous Engine to the next level and I figured I might as well present the results at CEDEC, leading to my topic.

―Could you give a brief explanation of the results of your research and testing?

Joudan:In my presentation, I talked about creating combat AI for an action game using a technique called deep learning. The focus of the research was how AI learns and becomes smarter all on its own through the simulations we run and I presented how I managed to create AI that seems as smart or even smarter and more natural than what could be made by hand without actually involving any manual work.

―You mentioned AI becoming smarter; how do you train AI and how does it get smarter specifically?

Joudan:We train AI by offering rewards for good behavior, such as for landing a successful attack or for successfully dodging an incoming attack. So, the AI eventually learns that it will be rewarded when certain conditions are met, and it will evolve into something that's closer to what we intend it to be.

―You mentioned "everyone is talking about deep learning" - is creating AI using deep learning becoming the norm in the game industry?

Joudan:I'm starting to see research being conducted, but I don't believe there is yet precedent of deep learning being effectively implemented into an actual product, which makes me want to implement it at our studio all the more. If AI can be fully created only by repeating simulations, it will help us cut down on cost considerably, and, depending on how you want to use it, I think this is something that can be attained in the very near future.


―While it has yet to be implemented in an actual product, is it nearly at that level?

Joudan:Yes, I think it could be implemented depending on the game design. I think the implementation itself won't be an issue if you simply expect the characters to seem somewhat intelligent and move smoothly, but it may be less compatible with games with strategic elements. Again, it all depends on how you want to use it.

―Do you think you will be able to implement the enemy UI created with deep learning in the project you're currently working on?

Joudan:It's hard to say at this point. It's up to the game specifications after all. We're still in the phase where we want AI to do what we intend it to do, so we will probably look into it more closely once the specifications are set.

―Do you think it would be difficult to move an enemy that follows a special, not as compatible, set of specifications with AI?

Joudan:That might be too difficult to do with deep learning, perhaps. Normally, boss characters have their own distinct patterns, but the disadvantage of deep learning is that it cannot make such distinction. So, I think the advantages of deep learning can be better highlighted and utilized on enemies that use simple movements and spawn in large quantities.

What are the benefits of using deep learning?

Joudan:Conventionally when creating AI there are various approaches such as scripting and using graphs, but whichever method you choose to go with, you are still burdened with the same degree of difficulty since you still need to list all possible behavioral patterns that the AI could perform. This requires logical thinking and is not something everyone can do, which makes it unfit for mass production. The selling point of deep learning is that the AI becomes smarter and smarter with repeated simulations, which leaves us humans with not much to do. The designers then can focus on other areas with the time they save, so I believe using deep learning can ultimately make the development of AI more efficient.



―As a result of your research, did you find a system that is unique and not found elsewhere for your presentation?

Joudan:When AI is created automatically, there are cases where the AI does not do what the designers intend it to do, which has been a major obstacle specifically for games. The approach we took as a countermeasure is called "Muchiuchi System", in other words "Whiplash System", which allows the designers to alter and adjust the result of what the system has learned to do. I think that is one of our selling points.

―What is the biggest advantage of designers being able to make adjustments afterwards?

Joudan:Normally, while you have some control through adjusting the rewards you give, it's hard to create "interesting" AI. However, by allowing the designers to make adjustments later, we can create AI with more personality, for instance cowardice or viciousness. This is something that is absolutely crucial in the coming era and the biggest advantage of this system.

―When playing and comparing the enemy created by AI with an enemy scripted by the designers, could you feel anything off with how the one created by AI behaved?

Joudan:No, we intentionally created it in a way that there wouldn't be much of a difference between the two. One acted not much differently from the other when we had a several different people play, so I'd say it's well made. (laugh)

The AI of tomorrow and showing its potential in a variety of use cases

―Your focus was on the enemy AI this time around - do you think you could also apply AI to the player effectively?

Joudan:This of course depends on the game systems as well, but if we were to apply deep learning to the player AI, the benefit would be that it can learn how each player prefers to control the character. By leveraging that data, we could perhaps create AI that imitates each player's play style and acts sort of like a mirror image of the player themselves, instead of the locked-in player AI created by the development team. That would be interesting and the players could perhaps feel a sense of attachment to the AI, too. Also, the same AI that has learned the player's play style could even possibly earn experience points all on its own.



―Is it correct that it can be applied to both the player and the enemy even if the character models are not humanoid?

Joudan:That is correct. We used a human character this time, but we actually did try it with beast-type enemies as well. The beast-type characters don't have intelligence to begin with, so when they're created with deep learning they act even more bestial. It made me feel that deep learning works better with beasts than it does with human characters. When a human model shows the slightest hint of strange movement, it instantly emphasizes its awkwardness and makes the model look less like a real human being, but that is not the case with beasts - we don't see much difference whether the beasts are moved by deep learning or by AI that has been handcrafted by us. As a result, even when they made some awkward moves, we often ended up liking them better.

―So, is it more difficult to reproduce something that actually exist in the real world like humans and animals?

Joudan:Yes, a certain amount of awkwardness is inevitable. Conversely, fantasy creatures that move unlike us real world creatures, tend to look more believable.

―What was the most challenging aspect of creating AI using deep learning?

Joudan:I would say the preparation of the environment and characters we were going to train. This time, we tried training a character made specifically for CEDEC use, so there were some concerns like "it might not work well depending on the used model" or "maybe we got good results last time just by luck," but to our surprise, it went smoothly and we ended up with more samples as a result, so things turned out well overall.


―Do you think, for example, online game bots, could potentially be replaced with AI created with deep learning moving forward?

Joudan:Yes, I think that is a possibility. Well, I don't think bots in general are capable of creating elaborate, high-quality AI, so yes, it is very possible that deep learning will replace them. I think it's better suited to create something that moves similar to a human being.

―The bots are widely used in competitive games, particularly battle royale type games - if the bots get replaced with deep learning, do you think the game can be played in a more natural way?

Joudan:I couldn't say for sure, but it'd be nice to see the enemy characters using AI moving in a very natural way or even imitating a player's actions, so that ultimately the players could get the feeling that they're playing with another human player.

―In your presentation, you discussed the enemy AI in battle, but could this also be applied to something like racing games?

Joudan:That should be possible, theoretically. We reward AI when training it, so as long as you can set the rewards right, it should work with racing games as well. That being said, terrain is very central in racing games and if we require data on that, it would increase the amount of learning. The more things the AI needs to take into account, the more complicated the issues become, so the AI not being able to fully learn everything is a concern.
When it comes to a racing game, the amount of information could be too much as the AI needs to consider where it currently is, where it is headed, where the walls are and such, whereas when it comes to combat, the AI only needs to concern itself with such things as the distance or relationship between itself and its opponent. It'd be really interesting if we could make it happen, but at this point, I couldn't say how it'd turn out without actually trying it.

―I see. I assume it is still in development, but when it finally becomes applicable to racing games, it'd be great to see an enemy AI that could elegantly merge the two priorities of reaching the goal and blocking the player, wouldn't it?

Joudan:I think we can realize enemies that have both behaviors, reaching the goal and interfering with its opponents, by equally weighting the rewards you give to the AI for both. However, it's also possible that the behavior ends up becoming equivocal. (laugh)

―What did you personally gain from your presentation?

Joudan:For me, the most significant thing was the fact that I used deep learning. The integration of deep learning will be the norm further down the road, I think. With that in mind, I got my hands on it a bit earlier and additionally was able to deliver actual results, so I personally, as well as the studio as a whole, made a significant step forward regardless of whether or not it will actually be implemented.
As such, setting the matter of implementation aside, the biggest advantage we gained as Luminous Productions is that we now have access to both options - to use deep learning or to create the enemy AI the conventional way.

―Would you say you now have more options at your disposal for creating a fun game?

Joudan:I agree that we have more options to choose from. It'd also be nice if we could create AI in large quantities using deep learning at the beginning of the development and with that have an environment where enemy AI is already moving about from the get go. That way, the designers can pour their effort into other elements that are relevant to improving playability. If they want to tweak AI by hand, they can always do that later.

―It seems to have a wide reaching positive impact. Are there any other technologies that seem feasible for application which you would personally want to look into for your research?

Joudan:There is a deep learning technique called GAN (Generative Adversarial Network), which generates a new image by imitating an existing image of someone. I had fun when I was studying it a bit, and in any case, it would be interesting to look into a different way of using deep learning.
The way of thinking in GAN is really interesting; the one who generates and the one who judges the authenticity of the generated image are both deep learning respectively. The system is set up in a way that each AI encourages the other to grow. It's not that I want to use this for something in particular, but rather, I just want to try it since it's such an interesting technique.

―In terms of future challenges, do you have a particular goal set for yourself?

Joudan:I've been given the opportunity to be the lead programmer on the project I'm working on right now; it's my first time being responsible for looking over the entire game, so first and foremost, I want to lead this project to completion successfully. Previously, I was only responsible for my own tasks, but now I have to look at the big picture.

―Do you have a mental image of the ideal lead programmer you wish to become someday?

Joudan:There is someone whom I respect and wish to become like him someday. He is kind, supportive of others and has such a great personality with quite thorough knowledge in programming - he is someone all young programmers aspire to be...he is good looking, too. (laugh) I'm working hard so I can hopefully become someone like him.


―What sort of studio is Luminous Productions to you, personally?

Joudan:The studio allows me the freedom to do what I like. The atmosphere here is great and it's very easy to work in. Of course, freedom isn't everything. (laugh) However, the studio does give us the opportunity to learn anything that might turn out to be interesting as part of our work, for which I'm grateful as not only does it helps me advance to the next step as an individual, but it's also good for the morale of the studio as a whole.


Facebook Share Twiiter Share Linkedin Share Reddit Share Tumblr Share