cwinter

2 ปีที่แล้วdiscuss

yes! the very last example in the post shows what happens if you truncate the mantissa to just the 4 most significant bits

cwinter

5 ปีที่แล้วdiscuss

Well if there was ever an appropriate time to plug my dark fantasy machine learning poem this is it: https://clemenswinter.com/2021/03/24/conjuring-a-codecraft-m...

cwinter

5 ปีที่แล้วdiscuss

The bot has actually never seen Destroyer's big drones during training even once, so I found it somewhat surprising that it even works as well as it does!

Completely agree that adding something like the "League" used by AlphaStar would be one of the top priorities if you wanted to push this project further. I don't think CodeCraft is sufficiently complex to really allow for several very distinct strategies in the same way as StarCraft II, but I would still expect training against a larger pool of more diverse agents to increase robustness quite a bit.

cwinter

5 ปีที่แล้วdiscuss

Thank you for the kind words! I am also quite excited about the new points in game design space that RL will unlock and am planning write another blogpost on that topic.

I quite like https://karpathy.github.io/2016/05/31/rl/ as an introduction to some of the ideas behind modern RL. Beyond that, I just recently found out about https://github.com/andyljones/reinforcement-learning-discord... which lists a lot of other high-quality resources.

CodeCraft is a programming game which you can "play" by writing a Scala/Java program that controls the game units. It's not actively developed anymore but still functional: http://codecraftgame.org/

cwinter

8 ปีที่แล้วdiscuss

Yes, excellent points. No doubt there is still much room for optimization.

cwinter

8 ปีที่แล้วdiscuss

The constraining factor in this case is CPU throughput. The CPU cannot execute instructions fast enough to keep up with the data being read in from memory.

Of course you are right that this is not the whole picture, it is possible to be bottlenecked on neither memory bandwidth nor CPU throughput.

With respect to memory, two additional considerations are read amplification and latency.

Read amplification: Read amplification basically means that you are reading data but not actually using it. Suppose we have a table stored in row format with four columns C1, C2, C3, C4 which store 8 byte integers. Row format means that the values for the columns are interleaved, so the memory layout would look like this, where cn refers to some value for column Cn:

  0x0000 c1 c2 c3 c4 c1 c2 c3 c4
  0x0040 c1 c2 c3 c4 c1 c2 c3 c4
  ...

One property of conventional memory is that we can only read memory in blocks of 64bytes (the size of a cache line). So if we have a loop that sums of all of the values in C1, we will actually also load all of the values for columns C2, C3, C4. This means we have a read amplification of factor 4, and our usable bandwidth will be a quarter of the theoretical maximum.

Latency: Suppose we also don't have any read amplification and make use of all values for each loaded cache line. Then after we finish processing a cache line, and load the next cache line, we might have to wait a long time (100s of cycles) before the new cache line actually arrives! So then we would actually be stalled because of memory latency during some parts of program execution.

One of the big advantages of using columnar layout (which stores all values for a column sequentially) is that it all but eliminates read amplification. Similarly, by accessing data sequentially we make it possible for the CPU to predict what data will be accessed in the future, and have it loaded into the cache before we even request it.

Of course there's a number of reasons why things might not work out quite so perfectly in practice, and those may why performance in this benchmark starts to degrade before the (usable) memory bandwidth reaches the theoretical maximum memory bandwidth. But as you can see it still possible to get very close, so it's safe to assume that queries which read much less data than the maximum bandwidth are constrained on CPU.

Submissions

The simple beauty of XOR floating point compression

Entity-Based Reinforcement Learning

Machine Learning and the Future of Video Games

Machine Learning and the Future of Video Games

A Brief Introduction to Nanothaumaturgy

The Pelko Method (2012)

Chinese AI lab challenges Google, OpenAI with a model of 1.75e12 parameters

Mastering Real-Time Strategy Games with Deep RL: Mere Mortal Edition

Building the Ultimate Deep Learning Workstation

How to Read 100s of Millions of Records per Second from a Single Disk

How to Read 100s of Millions of Records per Second from a Single Disk (2018)

How to Analyze Billions of Records per Second on a Single Desktop PC

comments