Re: [Vm-dev] [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

I'm mac osx

I will update the test library and ping back

On Mar 1, 2018, at 10:11 AM, Eliot Miranda eliot.miranda@gmail.com wrote:

Todd,

On Thu, Mar 1, 2018 at 10:01 AM, Eliot Miranda <eliot.miranda@gmail.com mailto:eliot.miranda@gmail.com> wrote: Hi Todd,

On Sun, Nov 26, 2017 at 8:24 AM, Todd Blanchard <tblanchard@mac.com mailto:tblanchard@mac.com> wrote:

i'm getting the idea that we should probably write a test suite/library for FFI

See the package FFI-Tests at http://source.squeak.org/FFI http://source.squeak.org/FFI and the file sqFFITestFuncs.c http://sqffitestfuncs.c/

I suggest you extend these with an example that demonstrates your failing case. One thing to be very careful about is the platform. Are you on Mac OS X, linux or Win64? Mac OS X & linux use the SysV ABI (see e.g. https://software.intel.com/sites/default/files/article/402129/mpx-linux64-ab... https://software.intel.com/sites/default/files/article/402129/mpx-linux64-abi.pdf) whereas Win64 uses Microsoft's own ABI (see https://msdn.microsoft.com/en-us/library/ms235286.aspx?f=255&MSPPError=-... https://msdn.microsoft.com/en-us/library/ms235286.aspx?f=255&MSPPError=-2147217396). In particular under the Sys V ABI certain structs (structs having two 64-bit fields) will be passed in registers, if two registers are available at that point in the parameter list. The ThreadedX64SysVFFIPlugin is written to pass structs in this manner but there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion. Microsoft's own ABI passes structs of size 8, 16, 32, or 64 bits and __m64 in a single register. The ThreadedX64Win64FFIPlugin is written to pass structs in this manner but again there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion.

HTH

...

On Nov 24, 2017, at 12:54 AM, Ben Coman <btc@openinworld.com mailto:btc@openinworld.com> wrote:

On 24 November 2017 at 13:16, Ben Coman <btc@openinworld.com mailto:btc@openinworld.com> wrote:

On 22 November 2017 at 21:59, Ben Coman <btc@openinworld.com mailto:btc@openinworld.com> wrote:

On 22 November 2017 at 13:38, Todd Blanchard <tblanchard@mac.com mailto:tblanchard@mac.com> wrote:

I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used... $ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation' if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)

I presume you meant... if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) so correcting and continuing...

$ clang test.c -L. -l microclang $ LD_LIBRARY_PATH=. ./a.out That failed

So I'm not sure how to proceed. I was expecting that would work while Pharo failed.

Now interestingly... $ clang test.c microlang.c $ ./a.out That worked

So it seems a similar problem exists outside our FFI.

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue. Attached zip exploded here for easy reference...

___microlang.h___ typedef unsigned uintptr_t;

typedef struct { const void *ptr_data[2]; } CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_(); int clang_getRangeEnd_(CXSourceRange_ range);

___microclang.c___ #include "microclang.h" const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_() { CXSourceRange_ range = {0}; range.ptr_data[0] = (void*)libraryString; return range; }

int clang_getRangeEnd_(CXSourceRange_ range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { return 0; } else { return 1; } }

___test.c___ #include <stdio.h> #include "microclang.h" const char* localString = "local_pointer";

void test( CXSourceRange_ range, char *note ) { int result = clang_getRangeEnd_(range); if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }

int main() { CXSourceRange_ range1 = clang_getArbitraryRange_(); test(range1, "library string");

    CXSourceRange_ range2 = {0};
    range2.ptr_data[0] = (void*)localString;
    test(range2, "local string");

}

___Makefile___ default: clean static shared

clean: rm -f *so *App @echo

shared: clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp @echo

static: clang -g -o staticApp test.c microclang.c ./staticApp @echo

Now running... $ make > report

gives... ___report___ rm -f *so *App

clang -g -o staticApp test.c microclang.c ./staticApp That worked (library string) That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp That failed (library string) That worked (local string)

Further simplification dealing *only* with strings (see attached sharedLibString.zip) (also attached is clang3.zip as a step along the way)

___microclang.c___ typedef unsigned uintptr_t; const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString() { return myLibraryString; }

int lib_testString( const char *aString ) { unsigned test = (uintptr_t)aString & 0x1; printf("\n test=%d, aString-->%d\n", test, (uintptr_t)aString); if (test) { return 0; } else { return 1; } }

___test.c___ #include <stdio.h> int lib_testString( const char *aString ); const char *lib_getLibraryString(); const char *localString = "local_pointer";

void test( int result, char *note ) { if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }

int main() { const char * libraryString = lib_getLibraryString(); test(lib_testString(libraryString), "library string");

    test(lib_testString(localString), "local string");

}

$ make > report

___report___ rm -f *so *App

clang -g -o staticApp test.c microclang.c ./staticApp

test=0, aString-->4196150 That worked (library string)

test=0, aString-->4196068 That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp

test=1, aString-->-792512599 That failed (library string)

test=0, aString-->4196484 That worked (local string)

cheers -ben

<sharedLibString.zip><clang3.zip>

-- _,,,^..^,,,_ best, Eliot

-- _,,,^..^,,,_ best, Eliot