Your log files indicate that your CPU usage is at 100% and your memory usage is just under 100% when you start trying to record so this is preventing FBX from getting going because there is nothing left for it to use.
If you have any unneeded apps running in the background, make sure to close those before trying to record to free up as much systems resources as possible, and if there are any game settings that you can trim back to reduce the strain on your PC, then you should give that a try to.