If the x264 encoder sends errors when FBX tries to access it, a temporary file will be left behind which will cause FBX to ignore the encoder.

If you have a C:\Users<Your username>\AppData\Roaming\Blueberry Software\FBXGameRecorder\vwfcodecproblem.flag file, rename that to something else and then try FBX again.

You should then see the x264 encoder but if there is a genuine problem with it, it may cause FBX to crash, in which case change the name of the temp file back and use a different encoder.