Archive for quitpatchingthings tag

Fixing 64 Bit Xilinx Apps Crashing Under Windows 8.1

Continue Reading

Several pieces of the Xilinx toolchain use the SmartHeap library. This library hooks allocator functions in msvcrt.dll by export – overwriting the original functions with jumps to their patched versions. This used to be fine, but now several of the allocator entrypoints on msvcrt are simply jumps to the real implementation. These are short jumps, so the instruction size count is small (typically 2-3 bytes). Add to that the 5 NOP bytes that usually pad functions apart and that isn’t enough for SmartHeap’s overwrite jumps. Here’s an example of the trampoline they write:

That’s 12 bytes – 4 too many. What I was seeing is the overwrite for operator new stomping over wcscmp which is right after it, and the application crashing later when calling this function:

You can see here how wcscmp has been overwritten by the tail of the patch. I did some searching around and found this post by a Xilinx employee, explaining how to turn off SmartHeap.

Basically the gist of it is that in the ISE_DS\ISE\lib\nt64 directory, back up libPortability.dll (which links to SmartHeap), and rename libPortabilityNOSH.dll to libPortability.dll. This fixed ISE for me, but unfortunately a lot of the other Xilinx tools have their own lib\nt64 directories with duplicate copies of libPortability.dll. For each lib\nt64 directory, you’ll need to overwrite libPortability.dll with that original libPortabilityNOSH.dll – which is not duplicated in the non-ISE lib\nt64 directories.

I checked for updates and it appears I’m running with the latest version (14.7) with no updates available. Hopefully Xilinx will distribute an update for the SmartHeap binaries.