If you have the 'Original' resolution setting selected in FBX, you will record at whatever resolution the game is running at. So if it is running at 1366x768, it will be recorded at 1366x768. It will not be reduced to 1280 x 720. In other words, you are not actually limited to recording at 720p by FBX.
Furthermore, you can select any of the other resolution options in FBX even if they exceed your screen's resolution. FBX will simply upscale or downscale from the game's resolution to fit the resolution you have selected. So you can create a 1080p or 1440p video if you want to - it will just be an upscaled version of your 1366x768 recording (and obviously won't look at good as if the game had really been running at those resolutions).
YouTube, on the other hand, insists on having very specific resolutions and will downscale anything that doesn't match one of these to the nearest one it does fit (it will never upscale). So anything recorded at 1366x768 would get downscaled to 1280 x 720 by YouTube. But if you have used the 1080p option in FBX the resolution will be 1920 x 1080 so YouTube would happily accept that as 1080p.
