Fixing 64 Bit Xilinx Apps Crashing Under Windows 8.1

Several pieces of the Xilinx toolchain use the SmartHeap library. This library hooks allocator functions in msvcrt.dll by export – overwriting the original functions with jumps to their patched versions. This used to be fine, but now several of the allocator entrypoints on msvcrt are simply jumps to the real implementation. These are short jumps, so the instruction size count is small (typically 2-3 bytes). Add to that the 5 NOP bytes that usually pad functions apart and that isn’t enough for SmartHeap’s overwrite jumps. Here’s an example of the trampoline they write:

That’s 12 bytes – 4 too many. What I was seeing is the overwrite for operator new stomping over wcscmp which is right after it, and the application crashing later when calling this function:

You can see here how wcscmp has been overwritten by the tail of the patch. I did some searching around and found this post by a Xilinx employee, explaining how to turn off SmartHeap.

Basically the gist of it is that in the ISE_DS\ISE\lib\nt64 directory, back up libPortability.dll (which links to SmartHeap), and rename libPortabilityNOSH.dll to libPortability.dll. This fixed ISE for me, but unfortunately a lot of the other Xilinx tools have their own lib\nt64 directories with duplicate copies of libPortability.dll. For each lib\nt64 directory, you’ll need to overwrite libPortability.dll with that original libPortabilityNOSH.dll – which is not duplicated in the non-ISE lib\nt64 directories.

I checked for updates and it appears I’m running with the latest version (14.7) with no updates available. Hopefully Xilinx will distribute an update for the SmartHeap binaries.

one Comment

  1. Dave On January 20, 2015 at 20:24

    A better fix is to use the SmartHeap libraries included with Vivado 2014.2 or newer.
    This forum post has more information.

Leave a comment

Your email address will not be published. Required fields are marked.

Latest Tweets