Log analysis and automated testing of the Luminous Engine for large-scale game development
―The theme of your presentation was about log analysis and automated testing – could you tell us what brought you to discuss this theme?
Iwasaki：Something we experienced often during the development of FINAL FANTASY XV was the occurrence of highly complicated bugs in which various factors were entwined due to the fact that not only was the game open world, but it was also such a large-scale project in terms of the content and the number of people involved. Our struggles with debugging continued well after the release of the core game, since it was operated as a live-service game with a patch released each month.
Based on that experience, it has become the task of the studio as a whole to explore a way to reduce the costs both on the Quality Assurance (QA) side where the testing is conducted, and on the development team side where the discovered bugs get fixed – this is in order to not only to save time, but to improve the quality of the game. This was the driving force behind our decision to focus our effort on “log analysis and automated testing of the Luminous Engine”, which was the theme of my presentation.
―The larger the scale of the title is, the larger the issue of QA cost reduction becomes. With that issue as the starting point, what led you to construct and test this particular system?
Iwasaki：Given the order of the development flow, we decided to first tackle the reduction of the debugging costs, rather than the costs on the QA side, where the test plays are conducted. At the time, we searched and searched for a system that is already out and available for use, thinking we didn’t need to build one from scratch ourselves, but we couldn’t find anything particularly remarkable. During our search my vision for the system we needed became clearer, so I figured we might as well build one ourselves, went ahead and implemented it and added a variety of functions along the way… and here we are!
―You introduced “Learning Replay” in your presentation – could you tell us about the different methods of “automated play” that are currently being used in the industry?
Iwasaki：Automated play and QA have become such hot topics these days, that there seems to be many ongoing projects at various companies. To name a couple of representative methods of automated play, there is a script-based method and a method using machine learning. The script-based method is something that has been used since back in the day, and it has been proven to produce steady results. However, the maintenance cost is quite high and it tends to be customized for a specific title, which makes it difficult to apply in a wide range of situations. On the other hand, the method using machine learning could potentially evolve into something very versatile, as research in the field of machine learning has been progressing rapidly in recent years. Particularly, the method where you only input game images is expected to be highly generic, and I think it has great potential. That being said, it is still in an experimental phase and my guess is that it would be difficult to utilize it for an actual production right away.
As such, while there are a number of methods that have been released with the aim to realize automated play, I believe there is yet to be one that can be called the “standard” method in terms of feasibility, maintenance costs, and general versatility. However, the demand for it within the industry is definitely high, so I’m pretty sure someone will come up with a standard that can be widely shared.
―Where would you position “Learning Replay” among these methods?
Iwasaki：In our search for a method that could potentially make the Luminous Engine usable for a variety of products, I personally felt that something that falls right between the script-based method and the machine learning method might be suitable. Thus the method called “Learning Replay”, which I presented at the conference, came to be.
Iwasaki：I should say I picked the mid-point between the script-based method and machine-learning method, in terms of feasibility and operational costs. The method is still under development, but I’m open to co-development opportunities if there are any companies that would be willing to collaborate in making the method a reality.
Or, if someone full of enthusiasm could join Luminous Productions and volunteer to develop it for us, I’d be more than happy to entrust the task to them immediately. (laugh)
―I see. So, you are seeking companies or people to co-develop the method with you. In order to co-develop such a system, what part of the development would you want to hand over to the other party?
Iwasaki：Let’s take a profession known as data scientist for example. They have highly specialized knowledge in mathematics required to analyze big data. However, while they are the specialists of data science, they do not have expertise that is unique to a particular field.
On the other hand, we are, of course, game development specialists, but are by no means data science experts. That being said, I don’t think a data scientist’s expertise alone can fully figure out “what sort of data should be collected as big data in order to develop a game?” and “what should we make the system learn?”
However, there are times when I wish I could leverage the expertise of the data scientists, particularly when I’m faced with a question like “what method should we use to make the machine learn the collected data?” In summary, I believe we can expect better improvements in automated play, if we tackle it from both the data scientist perspective and the game development one – so in other words, it’d be ideal if we could get someone to, for example, take on the part that requires calculations from the mathematics point of view, separately from the game development.
―Considering the seemingly increasing demands for technologies from various fields, would you agree that there will be more opportunities for professionals from outside the gaming industry to contribute?
Iwasaki： I would. As there are various methods of machine learning, it’d be ideal to work with and leverage the expertise of professionals from different fields. Needless to say, the field of machine learning is making drastic progress these days and something new comes out almost every month. To ensure we’re up to speed on all of these new updates and able to adopt them into our game development, I do wish to have a professional who specializes in that area by our side.
The evolution of efficient and high quality game development in a future where the use of automated play is practical
―When automated play and AI evolve enough that their use becomes practical in game development, do you think it will affect the order in which things are done in the game development process?
Iwasaki：Yes, I believe it will fluctuate depending on the technologies available at a given time, naturally. The automated QA process and easy-to-perform debugging alone can ensure the game is always in a releasable state.
For example, generally in game development, we pass the game on to QA toward the end of development, which means that the game almost always contains a number of bugs until then. If we could realize an environment where QA is run automatically and bugs can be removed at all times, that should allow us to create a stable version of the game basically any time, which means it could also be user tested more frequently – with that I believe the development method will evolve into one that can improve quality efficiently.
―So, in such a structure, would you have QA running each time a new build is created while still in the middle of development?
Iwasaki：That’s correct. In short, it would reduce the costs of releasing a game.
Till now, releasing a game has been a huge task that imposes great strain on the development team. Therefore, if that strain can be reduced, games could be released more frequently.
In fact, while the scale of the games vary, there are titles within the industry that have adopted a “games as a service” model, where they release the first chapter to the players as soon as it is created and develop the second and third chapters while receiving feedback from the players on the first chapter.
I think that makes a lot of sense. When the scale of the game reaches this level and the development costs get this high, it’d make more sense to create a part of the whole and release that part when it is done, instead of developing and releasing something big all at once – this way, we can hedge risks and listen to what our players have to say as well.
―So what is the factor that prevents you from releasing the game in such a style at this stage?
Iwasaki：If we were to release in that style the way we operate now, we would end up doing QA all year around. If we can cut down the QA and debugging costs, the “games as a service” release like you mentioned will probably become feasible, but at this point, it is highly unlikely for the development of a AAA scaled game based on the QA costs, which is why we are building a system to help us to cut down these costs.
―If such a live service release is realized, I assume the game quality will definitely improve?
Iwasaki：Yes, it will improve for sure. We should be able to create what we envision with ease that way, and if the game designers can play the finished product sooner, then the polishing work could start that much sooner as well.
―Being able to spend time fixing things that aren’t achieving the intended result or aren’t working on a particular element could potentially make the game more fun and lead to improved quality, I assume?
Iwasaki：Yes, when it comes to game development, the cost of spending time on unstable versions is extremely high; we spend so much time on fixing newly found bugs and other sudden issues. It’s crucial to fix bugs and errors, of course, but as a developer, I want to spend more time on creating something interesting than fixing issues.
岩崎氏：―Moving up the schedule so that the developers can release what reaches closer to their ideal, without having the players kept waiting – that certainly sounds like the dream
Objectives and visions for the future of large-scale game development
―In your presentation, you also mentioned the difficulties that large-scale game development entails - are these problems that you’ve personally had to continuously work on?
Iwasaki：Well, I’d say it’s more of the assignment of the industry as a whole than my own. It’s hard to describe the difficulties of large-scale game development in one word…but it’s often the case that something works out really well when it’s being handled by a limited number of people, but it suddenly starts falling apart as soon as it involves more hands. I believe there are many factors that affect this, for example, it is harder to track what work others are doing or grasp the overall state of the project when more and more people are involved.
With less people involved, it’s easy to see who is doing what and it’s also very clear what will be affected and how by your fixes. In addition, the schedule is easier to anticipate, and above all, the environment allows each and every one to be fully aware of the situation while they think and operate, so everyone can bring out 100% of their potential. However, things do not go quite like this in large-scale development… People tend to work in a segmented fashion, so when a major change needs to be made, we often encounter a number of difficulties due to the fact of the change being wider than anticipated – there’d be issues left and right.
―The AAA-scale game development that is taking place at Luminous Productions may be difficult to do with a smaller group of people – what sort of initiatives do you have in mind as a solution for that?
Iwasaki： This is actually something we’ve been incorporating right now. We’re taking what’s good in small-scale development, such as setting up task forces and creating small-size groups at the beginning, and integrating that into our development structure. That said, while the dev team is now split into smaller groups, the development environment itself still uses only one or several branches at most. As such, the branch remains much larger in scale compared to each group, so there are some remaining problems such as when a fix is made, the extent of its impact is still vast. In order to resolve such a situation, I think we should perhaps take an approach in which we split the branch by each team and integrate them later.
※Branching: in a game development management system, the duplication of an object under version control so that modifications can occur in parallel along multiple branches.
Iwasaki：However, if we go with that approach at this stage, each branch would end up evolving in different directions and it would require a lot of work to integrate them. What we need in order to avoid such a situation is a system that automatically merges and tests the contents of the branches in the background. If there is a mechanism for automated testing in place, I believe we can implement a method where the branch is split up for each team while we continue with our work as if it is small-scale development. And all the while, the integration testing is being automatically performed in the background, which will allow us to integrate only the elements that have been proved to be effective after testing. So, I think this will solve some of the difficulties of the development that involves a large group of people.
―Once such a system becomes possible, each small task team can pursue the best possible solution for their team without having to be too conscious about the main build, isn’t that right?
Luminous Productions is ever-evolving to ensure our development is always of the highest quality
―You mentioned the need to collaborate with professionals from the outside the game industry moving forward. Could you describe the ideal person you’d like to work with, given that they have interest in the development work at Luminous Productions?
Iwasaki： Someone who can identify their own tasks, with the ability to determine what needs to be done to solve them. As we are developing the engine ourselves, we need talents who are capable of thinking out of the box and look into what the root of the issue might be, such as “is this workflow appropriate for this engine to begin with?”, rather than putting efforts into learning how to use the engine and figuring out what to do within its limits.
―What kind of studio do you think Luminous Productions is, personally?
Iwasaki： I think the fact that we make our own engine alone makes us quite a rare studio.
For instance, while the automated testing is already proven to improve the development efficiency, I’d assume it would be difficult for other studios to dedicate their hours to work that is not directly linked to the product.
We can pour our time and effort into what could improve the efficiency of the development precisely because we create our own engine in our studio – that is definitely our advantage.
―What would you say is the biggest difference between a studio that develops their own engine and one that does not?
Iwasaki：The biggest difference, I believe, would be that the studio that is developing their own engine can work on and capitalize it with its general versatility in mind. Furthermore, what we’re creating with the engine is a AAA title for a brand new IP. As our aim is set high and we’re also developing the engine anticipating our future projects, it’s easier for us to challenge and invest costs into initiatives that may not be directly tied to the on-going game development. At other studios, the immediate game development has to be put higher on their priority list naturally, so I assume they often have to halt at some point and say “oh, we don’t have to go that far”, but that almost never happens at Luminous Productions.
―Is it the studio’s policy to proactively try out all ideas with future potential?
Iwasaki：Yes, I’d say so. Our aim is to materialize brand new ideas after all, so we have no intention whatsoever to create a carbon copy of a title that’s already been created by another company . We’re eager to engage in any technologies that seem promising, even if other companies haven’t picked up on it, and be bolder and edgier – that is the kind of mindset we have. And that’s not only me personally, but the studio as a whole.
―Even in the 1st installment of this interview series, the topic of making bold breakthroughs and continuing the challenge of creating the best possible AAA title popped up – it seems like everyone in the studio shares that same vision?
Iwasaki：To take on a challenge, even if it’s something that’s never been done before – that’s the kind of studio we are. Our eyes are on one thing and one thing only – to be bold and create something good.