Yes, really. It's fine for simple types but for more complex types in more complex situations, you pay the price.
Unique pointers carry around not just the type but also a default deleter, if you provide one. That deleter has to be copied around, checked for nullptr before execution and set to nullptr when ownership changes.
You are describing 'zero cost abstractions'. Except that they are anything but zero cost. Game developers and low level programmers shy away from them because of their impact in compile and build times, debug build performance and stack trace bloat, increased cognitive load, reduced refactoring ability. Even std::unique_ptr has runtime costs in release builds that a raw pointer does not.
In game development performance is a feature and anything that makes performance and memory usage less deterministic is frowned upon, like a GC. But iteration times are still paramount and things that make compile, load and link times longer and make it more difficult to debug are also frowned upon. Yes, we'd like to have our cake and eat it too :)
I would recommend you learn by doing, one step at a time.
Here is an example path that builds skills on top of each other, mostly geared towards programming, based on the background info you provided:
1. Create Tic Tac Toe. This will get you familiar with the core game loop. Read input, update game state, draw, repeat. It is a bounded project so you can finish it. Finishing projects is a skill on its own. If you want to use C or C++ have a look at Simple DirectMedia Layer (SDL) or Simple and Fast Multimedia Library (SFML). These libraries abstract OS functions like reading inputs and putting pixels on the screen so you can focus on the core game mechanics. If you want to use Python look at PyGame, for example. The industry mostly uses C++ with Visual Studio so keep that in mind if you want to join a company later on.
Here you can find some tutorials on SDL: http://lazyfoo.net/tutorials/SDL/index.php
2. Create Pong or Breakout or any such 2D game with familiar mechanics. Then asteroids. Then battle city. Then super mario land. Each of these on its own teaches you a skill that you didn't have before. If you find yourself repeating the same code, abstract it away. This is the beginning of a game engine - a small framework of reusable components from game to game.
3. Once you have done enough 2D projects that you feel like you have a good grip on most components individually and how they come together as a whole, move on to 3D. Read Game Engine Architecture. Read good open source game code like the doom3 http://fabiensanglard.net/doom3/index.php
Create small graphics demos where you can learn how to use shaders - shaderToy is helpful here. Create 3D Tetris.
4. By this point you will know enough about games to be dangerous. You will know what is most interesting to you. Maybe it's AI, or graphics, or physics, etc. Create demos and games that showcase that field you are really interested in. Try and partner up with people with complementary skills to yours.
There are many ways to learn game development, the one I'd recommend the most is the one that seems most interesting to you and gets you actually creating and finishing games.