﻿ Unreal Software - Thread: Learning / Evolving Bots | Machine Learning

# Forum

CS2D Scripts Learning / Evolving Bots | Machine Learning

# Learning / Evolving Bots | Machine Learning

22 replies
Page
1 2

#### Learning / Evolving Bots | Machine Learning

Loooser
User Offline

Some weeks ago I stumbled upon a youtube video by SethBling where a computer program "learns" how to play a level in Super Mario World. In this video he explains how he uses a process called neuroevolution to make the program finish the level.

So I started asking myself if I could write a program that "learns" how to play cs2d.

I took the code from SethBling and rewrote it so I could use it on cs2d bots. He was using NEAT (neuroevolution of augmenting topologies).

Before I continue, I recommend you to watch his video, where he also explains how it works:

How it works:
NEAT basically generates a fixed number of neural networks.

Neural networks are working similar to our brain. They get Inputs (information) and calculate Outputs (e.g. cs2d controls) based on the structure of the neural network.

All those generated neural networks get tested and rated with a fitness value based on how good they did on the given task (e.g killing enemies). When all neural networks have been tested the best with the highest fitness value get selected and a new generation of mutated neural networks gets generated. This new generation gets tested again -> best get selected -> new generation and so on...

Each generation the neural networks (should) get better.

Implementation:
When I finished rewriting the code I realized that it would be highly unrealistic to make bots fully controlled by neural nets.
Especially pathfinding is far to complex. So I decided to start with the enemy engage section of the bot code (when the bot has to fight an enemy).

Currently the neural network has the following inputs:
1 - Can I hit an enemy? Freeline and Angle (0 to 1 | no to yes)
2 - Nearest target angle (-1 to 1 | left to right)
3 - Target Distance (0 to 1 | near to far)
4 - Angle the enemy is aiming at relative me (-1 to 1 | left to right)

The neural network can give the following outputs:
1 - Intelligent Attack! (if output > 0.5)
2- Rotate (-1 to 1 | left to right)
3 - Change aim distance (0 to 1 | near to far)
4 - move X relative to target. (-1 to 1 | left to right)
5 - move Y relative to target. (-1 to 1 | go away to get closer)

fitness evaluation
:
Every time a bot takes longer than 200 frames to engage(timeout) or the bot respawns (bot has most likely died) the neural network gets evaluated and the next neural network gets tested.

fitness value is calculated with:
1 - Number of times the bot fires the right direction.
2 - Number of Scores / Kills
3 - Engagement time
4 - How close the bot was on aiming in the right direction

Here a video of a training process:
https://youtu.be/dk3VvSt6DXs

I also implemented a save an load functionality so that you don’t have to restart training every time the server gets restarted. It creates a save file after each generation.

Where to go from here?
I want to encourage everyone who is interested in trying it out and maybe make something cool or improve the current implementation.
Here is the Github Repository: https://github.com/LoooserDev/CS2D-NEAT-Bots/

It would be great if the results could be improved with better inputs and outputs. Maybe there is a potential to make great bots with this approach! Also a functionality to play the best neural network only is missing.

A visualisation of the neural network and stats on ai_debug screen would also be cool

Try different neural networks that also have memory cells. These could be useful for decision making that is not only based on the current frame. This would most likely be a Recurrent Neural Network. This is not so easy as it has to be implemented in a evolution process.

Make a great Video that hopefully gets many views to get more people interested in CS2D. SethBlings video has 4.5 mio. views and there are just a few videos on this topic. Also CS2D would be special since it’s a online shooter and real players can play against these Bots.
edited 1×, last 02.07.17 11:01:09 am

#### Re: Learning / Evolving Bots | Machine Learning

Starkkz
Moderator Offline

I was interested on this topic but I never got to know how to use correctly the back propagation.

#### Re: Learning / Evolving Bots | Machine Learning

Avo
User Offline

Wonderful to see someone made an attempt of making genetically evolving neural network for CS2D bots. Well done, sir.

OP has written
I want to encourage everyone who is interested in trying it out and maybe make something cool or improve the current implementation. (I will upload the files soon)
Repository would be good, especially if you want to find some helpers.

Starkkz has written
I never got to know how to use correctly the back propagation.
Same here.

#### Re: Learning / Evolving Bots | Machine Learning

_Yank
User Offline

Pretty cool! How big do the files get ?
I'd like too see the bots after 24 hours of training with actual players

What do you think of obversing the players ?

#### Re: Learning / Evolving Bots | Machine Learning

Loooser
User Offline

@ Starkkz: @ Avo: NEAT doesn't use back propagation it is generating mutations of the best neural nets. Every neural net has to be tested in game.

@ _Yank: The files are very small the save file is 1mb.

@ Avo: yes ill use a repository

#### Re: Learning / Evolving Bots | Machine Learning

KoshmaR
User Offline

An amazing job, the truth would be great this

#### Re: Learning / Evolving Bots | Machine Learning

Dousea
User Offline

Shit. I lost. This might be a better solution. Really want to see this happening.

#### Re: Learning / Evolving Bots | Machine Learning

_Yank
User Offline

@ Dousea: Of course not! I don't believe that this algorithm is going to surpass a well coded programmed A.I. especially because it would require lots of training and some compute power. I may be wrong though.

#### Re: Learning / Evolving Bots | Machine Learning

SQ
Moderator Offline

@ _Yank: Yeah, this bot would not come up with strategy and tactics. Performance wise also it's not the best solution due to that it's written in Lua. At the moment the worse part of the bots are the movement, they are always looking at the direction they are going and the patterns of all bots are the same. I thought of a quite challenging AI system, however I have no time for that.

Zeik
User Offline

Neat

#### Re: Learning / Evolving Bots | Machine Learning

GeoB99
Moderator Offline

Definitely this is amazing to see such a system like neuroevolution for CS2D bots. Out of curiosity, how's the performance like with the implementation of NEAT system in CS2D considering the complexity and the great amount of neural networks?

#### Re: Learning / Evolving Bots | Machine Learning

Pagyra
User Offline

Simple evolution is an ineffective selection of decisions and actions, it is not dependent on tactics, strategy, human behavior, player characteristics.
After all, in end we want to get bots that are similar and behave like a team of real live players, and not as terminators that destroy everything they see around them on their way to goal point.
Therefore, I think that you should try to combining existing code of regular Dousea's and old Loooser's bots (meaning those parts that help bot manage its character) and this new neural network but in that stage we need add some functions(human factors and parameters) to change final behavior, decisions and actions of bot.
May be we need a coders team to make correct scripts for bots.
And perhaps you need to add an analysis function of replays (or something like that), so that bot is trained based on data of real player's behavior.

#### Re: Learning / Evolving Bots | Machine Learning

Loooser
User Offline

Great to hear your opinions on this topic! I don't believe neuroevolution can replace the logic our current AI. But if improved neuroevolution has the opportunity to give the bots a more natural feel through its variation.

In order to improve the evolution speed I made every bot to test a different neural network. The benefit is also that the bots behave differently.

@ Dousea: Don't stop developing your bots. Im excited to see what you come up with. And maybe you can make use of this.

@ Pagyra: yes evolution takes a while to deliver good results. How good they get is determined by many factors inputs, outputs, fitness evaluation and how the evolution process is written. Simple "tactics" can be developed. But I agree an strategy or a team can't be developed. Thats way a mix of all would be great.

@ SQ: What kind of AI system where you thinking about?

@ GeoB99: As far as I can remember I had 31 bots running at the same time and the performance was good. Except for when the server had to generate a new generation of 300 neural networks and save it into a file, that's where the server lagged. But running neural networks without evolution should be fine from the performance perspective.

#### Re: Learning / Evolving Bots | Machine Learning

Dousea
User Offline

Well glad to hear. But if you don't post your code around this time, I wouldn't be able to make the bots compatible with this "necroevolution", but let's see what's ahead of us. Good luck with this.

#### Re: Learning / Evolving Bots | Machine Learning

sonkii
User Offline

At least the bots train themselves,not bots that doing the same mistakes.

#### Re: Learning / Evolving Bots | Machine Learning

xsiN
User Offline

Sometimes my WiFi crash for no reason and this could improve single player (since I'm forced to join it when I have got no WiFi)

#### Re: Learning / Evolving Bots | Machine Learning

User Offline

I like the idea (Machine Learning being currently my main field of study at university), however I do not believe learning to play cs2d is currently possible within a reasonable amount of time, I would imagine you need a crapload of runs before this might work well (thinking in the millions here, even though you got it down to a nice amount of useful parameters).

However, I wish you good luck, and I am looking forward to watching the process!

#### Re: Learning / Evolving Bots | Machine Learning

Pagyra
User Offline

Pagyra has written
And perhaps you need to add an analysis function of replays (or something like that), so that bot is trained based on data of real player's behavior.

I think that is really good idea for neuroevolution.

#### Re: Learning / Evolving Bots | Machine Learning

Avo
User Offline

Dousea has written
Shit. I lost. This might be a better solution. Really want to see this happening.

Hey, don't say anything about losing! Keep going with your AI, the time will show us which one is better. Or maybe in the end it will appear that synergy of your and Loooser's work will make an effective AI for CS2D bots.
1 2