Because FBX captures directly from the game itself, only the game's resolution is considered, not your desktop's.
If you have the 'Original' setting selected in FBX, the saved video will simply be the same resolution as the game is using. So if the game's resolution is 1080 x 1080, you should get an unstretched 1080 x 1080 video.
If you have 1080p, 720p, etc selected in FBX, the game's image will be scaled to that resolution but the aspect ratio will be maintained with black padding added to fill it out to the relevant 16:9 resolution. So if you had 1080p selected in FBX and the game is set to 1080 x 1080, I would expect that you would see the game's image 1080 x 1080 unstretched image with 420 pixels width of black on either side to give a 1920 x 1080 video.