English Learning / Evolving Bots | Machine Learning

22 replies
Goto Page
To the start Previous 1 2 Next To the start
Up
Loooser
User
Offline Off
Some weeks ago I stumbled upon a youtube video by SethBling where a computer program "learns" how to play a level in Super Mario World. In this video he explains how he uses a process called neuroevolution to make the program finish the level.

So I started asking myself if I could write a program that "learns" how to play cs2d.

I took the code from SethBling and rewrote it so I could use it on cs2d bots. He was using NEAT (neuroevolution of augmenting topologies).

Before I continue, I recommend you to watch his video, where he also explains how it works:
https://www.youtube.com/watch?v=qv6UVOQ0F44

> How it works:
NEAT basically generates a fixed number of neural networks.

Neural networks are working similar to our brain. They get Inputs (information) and calculate Outputs (e.g. cs2d controls) based on the structure of the neural network.

All those generated neural networks get tested and rated with a fitness value based on how good they did on the given task (e.g killing enemies). When all neural networks have been tested the best with the highest fitness value get selected and a new generation of mutated neural networks gets generated. This new generation gets tested again -> best get selected -> new generation and so on...

Each generation the neural networks (should) get better.

> Implementation:
When I finished rewriting the code I realized that it would be highly unrealistic to make bots fully controlled by neural nets.
Especially pathfinding is far to complex. So I decided to start with the enemy engage section of the bot code (when the bot has to fight an enemy).

Currently the neural network has the following inputs:
1 - Can I hit an enemy? Freeline and Angle (0 to 1 | no to yes)
2 - Nearest target angle (-1 to 1 | left to right)
3 - Target Distance (0 to 1 | near to far)
4 - Angle the enemy is aiming at relative me (-1 to 1 | left to right)

The neural network can give the following outputs:
1 - Intelligent Attack! (if output > 0.5)
2- Rotate (-1 to 1 | left to right)
3 - Change aim distance (0 to 1 | near to far)
4 - move X relative to target. (-1 to 1 | left to right)
5 - move Y relative to target. (-1 to 1 | go away to get closer)

fitness evaluation
:
Every time a bot takes longer than 200 frames to engage(timeout) or the bot respawns (bot has most likely died) the neural network gets evaluated and the next neural network gets tested.

fitness value is calculated with:
1 - Number of times the bot fires the right direction.
2 - Number of Scores / Kills
3 - Engagement time
4 - How close the bot was on aiming in the right direction

Here a video of a training process:
https://youtu.be/dk3VvSt6DXs

I also implemented a save an load functionality so that you don’t have to restart training every time the server gets restarted. It creates a save file after each generation.

>Where to go from here?
• I want to encourage everyone who is interested in trying it out and maybe make something cool or improve the current implementation.
Here is the Github Repository: https://github.com/LoooserDev/CS2D-NEAT-Bots/

• It would be great if the results could be improved with better inputs and outputs. Maybe there is a potential to make great bots with this approach! Also a functionality to play the best neural network only is missing.

• A visualisation of the neural network and stats on ai_debug screen would also be cool

• Try different neural networks that also have memory cells. These could be useful for decision making that is not only based on the current frame. This would most likely be a Recurrent Neural Network. This is not so easy as it has to be implemented in a evolution process.

• Make a great Video that hopefully gets many views to get more people interested in CS2D. SethBlings video has 4.5 mio. views and there are just a few videos on this topic. Also CS2D would be special since it’s a online shooter and real players can play against these Bots.
edited 1×, last 02.07.17 11:01:09 am
IMG:http://i45.tinypic.com/saus1e.jpg
02.07.17 12:51:01 am
Up
Starkkz
Moderator
Offline Off
I was interested on this topic but I never got to know how to use correctly the back propagation.
lol
02.07.17 12:54:14 am
Up
Avo
User
Offline Off
Wonderful to see someone made an attempt of making genetically evolving neural network for CS2D bots. Well done, sir.

OP has written:
I want to encourage everyone who is interested in trying it out and maybe make something cool or improve the current implementation. (I will upload the files soon)
Repository would be good, especially if you want to find some helpers.

user Starkkz has written:
I never got to know how to use correctly the back propagation.
Same here.
Trust me, I'm an engineer | user DC approved file cs2d Super extra mod for CS2D (64), yeah!
02.07.17 01:01:19 am
Up
_Yank
User
Offline Off
Pretty cool! How big do the files get ?
I'd like too see the bots after 24 hours of training with actual players

What do you think of obversing the players ?
02.07.17 01:08:07 am
Up
Loooser
User
Offline Off
@user Starkkz: @user Avo: NEAT doesn't use back propagation it is generating mutations of the best neural nets. Every neural net has to be tested in game.

@user _Yank: The files are very small the save file is 1mb.

@user Avo: yes ill use a repository
IMG:http://i45.tinypic.com/saus1e.jpg
02.07.17 01:15:12 am
Up
KoshmaR
User
Offline Off
An amazing job, the truth would be great this
02.07.17 02:06:31 am
Up
Ortimh
User
Offline Off
Shit. I lost. This might be a better solution. Really want to see this happening.
I'm awesome ... and I really like cookies.
02.07.17 02:11:08 am
Up
_Yank
User
Offline Off
@user Ortimh: Of course not! I don't believe that this algorithm is going to surpass a well coded programmed A.I. especially because it would require lots of training and some compute power. I may be wrong though.
02.07.17 02:13:43 am
Up
SQ
Moderator
Offline Off
@user _Yank: Yeah, this bot would not come up with strategy and tactics. Performance wise also it's not the best solution due to that it's written in Lua. At the moment the worse part of the bots are the movement, they are always looking at the direction they are going and the patterns of all bots are the same. I thought of a quite challenging AI system, however I have no time for that.
02.07.17 04:49:07 am
Up
Zeik
User
Offline Off
Neat
02.07.17 09:37:09 am
Up
GeoB
Moderator
Offline Off
Definitely this is amazing to see such a system like neuroevolution for CS2D bots. Out of curiosity, how's the performance like with the implementation of NEAT system in CS2D considering the complexity and the great amount of neural networks?
02.07.17 09:39:48 am
Up
Pagyra
User
Offline Off
Simple evolution is an ineffective selection of decisions and actions, it is not dependent on tactics, strategy, human behavior, player characteristics.
After all, in end we want to get bots that are similar and behave like a team of real live players, and not as terminators that destroy everything they see around them on their way to goal point.
Therefore, I think that you should try to combining existing code of regular user Ortimh's and old user Loooser's bots (meaning those parts that help bot manage its character) and this new neural network but in that stage we need add some functions(human factors and parameters) to change final behavior, decisions and actions of bot.
May be we need a coders team to make correct scripts for bots.
And perhaps you need to add an analysis function of replays (or something like that), so that bot is trained based on data of real player's behavior.
02.07.17 10:24:47 am
Up
Loooser
User
Offline Off
Great to hear your opinions on this topic! I don't believe neuroevolution can replace the logic our current AI. But if improved neuroevolution has the opportunity to give the bots a more natural feel through its variation.

In order to improve the evolution speed I made every bot to test a different neural network. The benefit is also that the bots behave differently.

@user Ortimh: Don't stop developing your bots. Im excited to see what you come up with. And maybe you can make use of this.

@user Pagyra: yes evolution takes a while to deliver good results. How good they get is determined by many factors inputs, outputs, fitness evaluation and how the evolution process is written. Simple "tactics" can be developed. But I agree an strategy or a team can't be developed. Thats way a mix of all would be great.

@user SQ: What kind of AI system where you thinking about?

@user GeoB: As far as I can remember I had 31 bots running at the same time and the performance was good. Except for when the server had to generate a new generation of 300 neural networks and save it into a file, that's where the server lagged. But running neural networks without evolution should be fine from the performance perspective.
IMG:http://i45.tinypic.com/saus1e.jpg
02.07.17 10:33:02 am
Up
Ortimh
User
Offline Off
Well glad to hear. But if you don't post your code around this time, I wouldn't be able to make the bots compatible with this "necroevolution", but let's see what's ahead of us. Good luck with this.
I'm awesome ... and I really like cookies.
02.07.17 11:02:22 am
Up
Loooser
User
Offline Off
@user Ortimh: I have uploaded the files in a GitHub repository.

https://github.com/LoooserDev/CS2D-NEAT-Bots/
IMG:http://i45.tinypic.com/saus1e.jpg
02.07.17 11:50:35 am
Up
PeculiarLookingLink
User
Offline Off
At least the bots train themselves,not bots that doing the same mistakes.
Inactive dude
02.07.17 12:25:00 pm
Up
xsiN
User
Offline Off
Sometimes my WiFi crash for no reason and this could improve single player (since I'm forced to join it when I have got no WiFi)
Well done, please continue it
ignore my previous names
02.07.17 03:26:20 pm
Up
Hador
User
Offline Off
I like the idea (Machine Learning being currently my main field of study at university), however I do not believe learning to play cs2d is currently possible within a reasonable amount of time, I would imagine you need a crapload of runs before this might work well (thinking in the millions here, even though you got it down to a nice amount of useful parameters).

However, I wish you good luck, and I am looking forward to watching the process!
02.07.17 03:43:42 pm
Up
Pagyra
User
Offline Off
user Pagyra has written:
And perhaps you need to add an analysis function of replays (or something like that), so that bot is trained based on data of real player's behavior.

I think that is really good idea for neuroevolution.
02.07.17 07:46:00 pm
Up
Avo
User
Offline Off
user Ortimh has written:
Shit. I lost. This might be a better solution. Really want to see this happening.

Hey, don't say anything about losing! Keep going with your AI, the time will show us which one is better. Or maybe in the end it will appear that synergy of your and user Loooser's work will make an effective AI for CS2D bots.
Trust me, I'm an engineer | user DC approved file cs2d Super extra mod for CS2D (64), yeah!
To the start Previous 1 2 Next To the start