Several pieces of the Xilinx toolchain use the SmartHeap library. This library hooks allocator functions in msvcrt.dll by export – overwriting the original functions with jumps to their patched versions. This used to be fine, but now several of the allocator entrypoints on msvcrt are simply jumps to the real implementation. These are short jumps, so the instruction size count is small (typically 2-3 bytes). Add to that the 5 NOP bytes that usually pad functions apart and that isn’t enough for SmartHeap’s overwrite jumps. Here’s an example of the trampoline they write:
000000000013b220 48b85079910a00000000 mov rax,offset SHSMP64!shi_new (00000000`0a917950)
000000000013b22a ffe0 jmp rax
That’s 12 bytes – 4 too many. What I was seeing is the overwrite for operator new stomping over wcscmp which is right after it, and the application crashing later when calling this function:
0:000> uf msvcrt!wcscmp
Flow analysis was incomplete, some code may be missing
00007ffcd1564694 0000 add byte ptr [rax],al
00007ffcd1564696 ffe0 jmp rax
You can see here how wcscmp has been overwritten by the tail of the patch. I did some searching around and found this post by a Xilinx employee, explaining how to turn off SmartHeap.
Basically the gist of it is that in the ISE_DS\ISE\lib\nt64 directory, back up libPortability.dll (which links to SmartHeap), and rename libPortabilityNOSH.dll to libPortability.dll. This fixed ISE for me, but unfortunately a lot of the other Xilinx tools have their own lib\nt64 directories with duplicate copies of libPortability.dll. For each lib\nt64 directory, you’ll need to overwrite libPortability.dll with that original libPortabilityNOSH.dll – which is not duplicated in the non-ISE lib\nt64 directories.
I checked for updates and it appears I’m running with the latest version (14.7) with no updates available. Hopefully Xilinx will distribute an update for the SmartHeap binaries.