So my projects look like this:
I linked Assimp inside of Core. There I created a class that includes the assimp headers.
Game links Core.
Now if I create an object in Game of the class that uses the assimp header, it says that it can't find the header files.
So how to make it work, so I don't have to link assimp within game?
There is a difference between LINKING a static library, and INCLUDING the headers for a project: C++ : Difference between linking library and adding include directories
You say you've linked assimp inside of core but that's not possible, a static library needs to be linked into the executable that uses it, however they cannot link other static libraries. I'm guessing you mean you have included the headers for assimp in the core project which is correct, however if you have exposed any of the assimp API through the core project you will then also need to include the assimp headers in any other project which uses core. If you want to avoid doing this you can hide the assimp API by only including it's headers in .cpp files.
The game project will also need to link BOTH core and assimp regardless as both need to be linked into the final executable,
What Microsoft has to say about using static libraries: https://msdn.microsoft.com/en-us/library/ms235627.aspx
Brief example to clarify:
Core project header file, core.h
Game header file, game.h
Now when core.h is included through the game project, assimp.h is also included, this means the game needs to know the location of assimp.h