On 24 March 2011 19:18, Eliot Miranda eliot.miranda@gmail.com wrote:

etting you and others to start pounding on these would be fantastic.  Just a matter of finding time.  f you're happy for me to push inbox packages, changesets, workspace contents then I can provide something quickly.  If you want me to commit canges that I've made sure are backward compatible and don't break trunk I'm going ot be much slower.

Another alternative is to create a gitorious account and publish your intermediate hacks there, and then once you ready, commit to blessed repository. In this way you can make anything available, without endangering main branch :)

And don't worry, we have a client for you who can test new functionality deeply - an Ocean project which implements networking stack from scratch and it badly needs threaded FFI.

best regards (albeit feeling a little frazzled :-) ), Eliot

On 24 March 2011 20:25, Mariano Martinez Peck marianopeck@gmail.com wrote:

On Thu, Mar 24, 2011 at 7:27 PM, Igor Stasenko siguctua@gmail.com wrote:

...

On 24 March 2011 19:18, Eliot Miranda eliot.miranda@gmail.com wrote:

...

etting you and others to start pounding on these would be fantastic.  Just a matter of finding time.  f you're happy for me to push inbox packages, changesets, workspace contents then I can provide something quickly.  If you want me to commit canges that I've made sure are backward compatible and don't break trunk I'm going ot be much slower.

Another alternative is to create a gitorious account and publish your intermediate hacks there, and then once you ready, commit to blessed repository. In this way you can make anything available, without endangering main branch :)

And don't worry, we have a client for you who can test new functionality deeply - an Ocean project which implements networking stack from scratch and it badly needs threaded FFI.

Don't forget the new team for DBXTalk ;)

Yeah... so Eliot, as you can see you have customers lined up at your office already :)

...

...

best regards (albeit feeling a little frazzled :-) ), Eliot

-- Best regards, Igor Stasenko AKA sig.

On Thu, 24 Mar 2011, Eliot Miranda wrote:

(pine can't quote your mail, sorry)

"I have some image-level changes that I'll try and commit asap, or at least give you as some inbox packages - parsing FFI method annotations to discover the #threaded keyword that sets the "go ahead and thread this call" bit in an FFI spec - an additional inst var in ExternalFunction (IIRC) that allows the VM to record how much stack space to reserve when marshalling an FFI call - the new callback marshalling machinery that provides platform-independence

I just need to fid the time to push the code to you.

Now on testing I use a) an image with a "listener" that reads and writes from/to stdin/out while allowing one to interact with the image b) a native Mac OS file dialog that uses a threaded call to invoke without blocking the image and threaded callbacks to determine which files to show (filter callback) and what the accept action is.  I've of course lost my workspace containing this and so have to ferret out the doits from my changes file (I feel such a fool!).  Alas this will need work as it used an extension to a Teleplace native file dialog plugin.  But posting it to FFI will be good; it will test the FFI further. c) a threaded version of the ODBC connect that hasn't really been tested

etting you and others to start pounding on these would be fantastic.  Just a matter of finding time.  f you're happy for me to push inbox packages, changesets, workspace contents then I can provide something quickly.  If you want me to commit canges that I've made sure are backward compatible and don't break trunk I'm going ot be much slower.

best regards (albeit feeling a little frazzled :-) ), Eliot"

Isn't it the best time to migrate the syntax of FFI calls to Pragmas? I know it takes some time to implement the pragma support, but IIRC Lukas did that a few years ago, so dusting it off and adding support for threaded calls (which I didn't see yet) shouldn't be that hard. Both formats could be supported for a while, then we could leave the current format behind and simplify the parsers/compilers, etc.

Levente

On 24 March 2011 19:43, Eliot Miranda eliot.miranda@gmail.com wrote:

2011/3/24 Levente Uzonyi leves@elte.hu

...

On Thu, 24 Mar 2011, Eliot Miranda wrote:

(pine can't quote your mail, sorry)

:)  No problem.

...

"I have some image-level changes that I'll try and commit asap, or at least give you as some inbox packages

• parsing FFI method annotations to discover the #threaded keyword that sets the "go ahead and thread this call" bit in an FFI spec
• an additional inst var in ExternalFunction (IIRC) that allows the VM to record how much stack space to reserve when marshalling an FFI call
• the new callback marshalling machinery that provides platform-independence

I just need to fid the time to push the code to you.

Now on testing I use a) an image with a "listener" that reads and writes from/to stdin/out while allowing one to interact with the image b) a native Mac OS file dialog that uses a threaded call to invoke without blocking the image and threaded callbacks to determine which files to show (filter callback) and what the accept action is.  I've of course lost my workspace containing this and so have to ferret out the doits from my changes file (I feel such a fool!).  Alas this will need work as it used an extension to a Teleplace native file dialog plugin.  But posting it to FFI will be good; it will test the FFI further. c) a threaded version of the ODBC connect that hasn't really been tested

etting you and others to start pounding on these would be fantastic.  Just a matter of finding time.  f you're happy for me to push inbox packages, changesets, workspace contents then I can provide something quickly.  If you want me to commit canges that I've made sure are backward compatible and don't break trunk I'm going ot be much slower.

best regards (albeit feeling a little frazzled :-) ), Eliot"

Isn't it the best time to migrate the syntax of FFI calls to Pragmas? I know it takes some time to implement the pragma support, but IIRC Lukas did that a few years ago, so dusting it off and adding support for threaded calls (which I didn't see yet) shouldn't be that hard.

+1000.

SPunds good.  Got any URLs?  Lukas, got any code?

...

Both formats could be supported for a while, then we could leave the current format behind and simplify the parsers/compilers, etc.

+1

Indeed.  In fact, nothing is set in stone here.  I desperately want an accurate C signature, not the quick hack we have now (a good thing we have a quick hack, but its a long way short of copy/pasting C into the pragma).  The interpreted marshalling scheme Andreas did (ffi specs etc) can serve for the moment.  The main thing before getting rid of that is a design for e.g. a NativeBoost based marshalling code generator that is cross-platform, cross-ABI and works with the interpreted VMs.  I think this is therefore a good way off, and there are people like Esteban who need to get going now, so we can't let the perfect be the enemy of the good and need to provide something functional asap, and that may mean living with the current ffi annotation for a few months.

In NativeBoost i made a separate class for parsing a C function declaration. It supports two forms of declaration: - named function, i.e. int foo (int bar, float zork) and anonymous one: int (int bar, float zork)

as input, parser takes a literal array.

So its quite easy to integrate it into compiler/pragma parser, we only need to find out what format will work for us.

so, it could be something like:

<cdecl: #( int foo (int bar , float baz )) module: #myModule>

or: <ffi: #( cdecl int foo ... ) module: ... >

The class for parsing function prototype can be found in SqS/NativeBoost in class NBFnSpecParser and tests are in: NBFnSpecParserTest in NativeBoost-Test package. (some of them are NB-specific) but its easy to rip out extra functionality :)

Also, there is a wiki page explaining this format: http://code.google.com/p/nativeboost/wiki/NBFnSpecParser

So, if we agree with plan, i could port this class and adopt it for compiler (and yes , by taking Lukas code as well)

On 24.03.2011 20:10, Eliot Miranda wrote:

On Thu, Mar 24, 2011 at 12:02 PM, Igor Stasenko <siguctua@gmail.com mailto:siguctua@gmail.com> wrote:

On 24 March 2011 19:43, Eliot Miranda <eliot.miranda@gmail.com
<mailto:eliot.miranda@gmail.com>> wrote:
>
>
>
> 2011/3/24 Levente Uzonyi <leves@elte.hu <mailto:leves@elte.hu>>
>>
>>
>> On Thu, 24 Mar 2011, Eliot Miranda wrote:
>>
>> (pine can't quote your mail, sorry)
>
> :)  No problem.
>
>>
>> "I have some image-level changes that I'll try and commit asap,
or at least give you as some inbox packages
>> - parsing FFI method annotations to discover the #threaded
keyword that sets the "go ahead and thread this call" bit in an
FFI spec
>> - an additional inst var in ExternalFunction (IIRC) that allows
the VM to record how much stack space to reserve when marshalling
an FFI call
>> - the new callback marshalling machinery that provides
platform-independence
>>
>> I just need to fid the time to push the code to you.
>>
>> Now on testing I use
>> a) an image with a "listener" that reads and writes from/to
stdin/out while allowing one to interact with the image
>> b) a native Mac OS file dialog that uses a threaded call to
invoke without blocking the image and threaded callbacks to
determine which files to
>> show (filter callback) and what the accept action is.  I've of
course lost my workspace containing this and so have to ferret out
the doits from my
>> changes file (I feel such a fool!).  Alas this will need work
as it used an extension to a Teleplace native file dialog plugin.
 But posting it to
>> FFI will be good; it will test the FFI further.
>> c) a threaded version of the ODBC connect that hasn't really
been tested
>>
>> etting you and others to start pounding on these would be
fantastic.  Just a matter of finding time.  f you're happy for me
to push inbox packages,
>> changesets, workspace contents then I can provide something
quickly.  If you want me to commit canges that I've made sure are
backward compatible
>> and don't break trunk I'm going ot be much slower.
>>
>> best regards (albeit feeling a little frazzled :-) ),
>> Eliot"
>>
>> Isn't it the best time to migrate the syntax of FFI calls to
Pragmas? I know it takes some time to implement the pragma
support, but IIRC Lukas did that a few years ago, so dusting it
off and adding support for threaded calls (which I didn't see yet)
shouldn't be that hard.
>
+1000.

> SPunds good.  Got any URLs?  Lukas, got any code?
>
>>
>> Both formats could be supported for a while, then we could
leave the current format behind and simplify the
parsers/compilers, etc.
>
+1

> Indeed.  In fact, nothing is set in stone here.  I desperately
want an accurate C signature, not the quick hack we have now (a
good thing we have a quick hack, but its a long way short of
copy/pasting C into the pragma).  The interpreted marshalling
scheme Andreas did (ffi specs etc) can serve for the moment.  The
main thing before getting rid of that is a design for e.g. a
NativeBoost based marshalling code generator that is
cross-platform, cross-ABI and works with the interpreted VMs.  I
think this is therefore a good way off, and there are people like
Esteban who need to get going now, so we can't let the perfect be
the enemy of the good and need to provide something functional
asap, and that may mean living with the current ffi annotation for
a few months.

In NativeBoost i made a separate class for parsing a C function
declaration. It supports two forms of declaration:
 - named function, i.e.
  int foo (int bar, float zork)
and anonymous one:
 int (int bar, float zork)

as input, parser takes a literal array.

So its quite easy to integrate it into compiler/pragma parser, we only
need to find out what format will work for us.

so, it could be something like:

<cdecl: #( int foo (int bar , float baz )) module: #myModule>

or:
<ffi: #( cdecl int foo ... ) module: ... >

This is such a fascinating hack. e.g.

#(int (*compar)(const void *, const void *)) #(#int #(#* #compar) #(#const #void #*, #const #void #*))

#(int (struct { field : 8; fence : 8; gate: 16; } arg)) #(#int #(#struct #'{' #field #':' 8 #';' #fence #':' 8 #';' #gate: 16 #';' #'}' #arg))

Can anyone prove that Igor's approach works for all C signatures? What about other languages?

I find this is like really dirty sex, repellent and irresistible at one and the same time.

Thank you for putting words to my feeling at the time I suggested it :) Don't know if it works for every signature though, only the examples he gave of the old format :/

Cheers, Henry

On Thu, Mar 24, 2011 at 1:10 PM, Lukas Renggli renggli@gmail.com wrote:

...

...

Isn't it the best time to migrate the syntax of FFI calls to Pragmas? I

know it takes some time to implement the pragma support, but IIRC Lukas did that a few years ago, so dusting it off and adding support for threaded calls (which I didn't see yet) shouldn't be that hard.

...

SPunds good. Got any URLs? Lukas, got any code?

Sorry, I don't have that code anymore. I used the existing FFI parser code within a string literal in the pragma.

So what do you think of the literal array hack. And remember, this is for <goog_1321969675> *posterity <goog_1321969675>* so <goog_1321969675>*be honest<goog_1321969675> *. http://www.imdb.com/title/tt0093779/quotes

Lukas

-- Lukas Renggli www.lukas-renggli.ch

On 24 March 2011 21:50, Lukas Renggli renggli@gmail.com wrote:

...

...

...

...

Isn't it the best time to migrate the syntax of FFI calls to Pragmas? I know it takes some time to implement the pragma support, but IIRC Lukas did that a few years ago, so dusting it off and adding support for threaded calls (which I didn't see yet) shouldn't be that hard.

SPunds good.  Got any URLs?  Lukas, got any code?

Sorry, I don't have that code anymore. I used the existing FFI parser code within a string literal in the pragma.

So what do you think of the literal array hack.  And remember, this is for posterity so be honest.

I don't like it to have methods polluted with meaningless symbols (that will also show up in senders/implementors).

That's exactly why i like it. Now i could use standard tools to figure out which FFI function using some fancy type(s). And that's exactly why i using symbols in primitive names:

<primitive: #primName module: #moduleName>

So, i can quickly access them. And it is _not_ meaningless. Its handy.

I doubt that the symbol hack will make parsing much simpler, because you still need to analyze the token sequence.

Yes, that's what my class does. It is reusing smalltalk parser as a tokenizer.

Lukas

-- Lukas Renggli www.lukas-renggli.ch

So, what is the outcome of discussion?

Are we agree to use a literal array in new syntax? And of course, we will preserve the old syntax for the time being.

If yes, then we should continue discussing, what new pragmas (and in what form) will use new syntax.

On 25 March 2011 19:03, Colin Putney colin@wiresong.com wrote:

Looks like a bit of interpretation is necessary here.

Lukas wrote:

...

...

I don't like it to have methods polluted with meaningless symbols (that will also show up in senders/implementors).

Igor replied:

...

That's exactly why i like it. Now i could use standard tools to figure out which FFI function using some fancy type(s). And that's exactly why i using symbols in primitive names:

<primitive: #primName module: #moduleName>

So, i can quickly access them. And it is _not_ meaningless. Its handy.

Igor is right, using named primitives make it easy to find out where primitives are called, since they show up as senders. However, consider Eliot's example from earlier in the thread:

#(int (struct { field : 8; fence : 8; gate: 16; } arg))

Here you get a lot more than the module name and the primitive name. You also end up with a bunch of C keywords, identifiers and punctuation in the symbol table:

#':' #';' #'{' #'}'

This is really extreme example just to see if it can parse things like above. Normally you won't find declarations like that. And after all, you can always do some cleanup :)

#arg #fence #field #gate #int #struct

That *is* pretty meaningless. You can get spurious senders of things like #field and #gate, but at the same time, you can't easily search for FFI calls that include 'struct { field : 8; fence : 8; gate: 16; }' in them.

You can always rename your arguments to something unique, to ensure that only your methods using them in signatures. Then you can make use of browser to help you finding stuff quickly. Really. It is the same as with usual code. If you name your method like #value , then it is your fault that you cannot meaningfully (or quickly) find all senders which will invoke your method.

So i found this as a weak argument against this syntax, because it applies to the rest of smalltalk code as well.

...

...

I doubt that the symbol hack will make parsing much simpler, because you still need to analyze the token sequence.

Yes, that's what my class does.  It is reusing smalltalk parser as a tokenizer.

Lukas' point is that this hack doesn't actually buy us much. We get a scanner for free, but that's the easy part. The hard part is implementing the parser.

If the point of all this is to take advantage of the fact that Igor has already done the hard work of implementing a parser for C declarations, why not choose a cleaner syntax, write a scanner for it, and adapt Igor's parser to fit?

I mean, we already have a quick-and-dirty syntax. It works fine. If we're going to replace it, we'll have to maintain backward compatibility during the transition period, deprecate the old syntax, encourage/help people migrate to the new syntax etc. Do we really want to do that for something that still doesn't get us proper C declaration pragmas?

Note that in NativeBoost i'm not using pragmas for defining a function signature:

destroyWindow "destroy the window" <primitive: 'primitiveNativeCall' module: 'NativeBoostPlugin'>

^ NBFFICallout stdcall: #( BOOL DestroyWindow ( HWND self )) module: #user32 options: #( + optMayCallback ) " calls windowproc"

So, yes, i could put everything into string literal.. no big deal:

stdcall: ' BOOL DestroyWindow ( HWND self ) ' instead of: stdcall: #( BOOL DestroyWindow ( HWND self ))

but then i cannot browse all senders of BOOL, DestroyWindow,HWND anymore :(

Colin

On 3/24/2011 19:35, Levente Uzonyi wrote:

Isn't it the best time to migrate the syntax of FFI calls to Pragmas? I know it takes some time to implement the pragma support, but IIRC Lukas did that a few years ago, so dusting it off and adding support for threaded calls (which I didn't see yet) shouldn't be that hard.

Sorry, but that's a decidedly bad idea. It breaks each and every use of the FFI out there. The FFI syntax and support is what it is, so please leave it alone. If you want to implement something different, then implement something different and do it in a form that doesn't conflict with the FFI so that both can live side by side.

Cheers, - Andreas

Both formats could be supported for a while, then we could leave the current format behind and simplify the parsers/compilers, etc.

Levente

On 3/25/2011 7:18, Levente Uzonyi wrote:

On Fri, 25 Mar 2011, Andreas Raab wrote:

...

On 3/24/2011 19:35, Levente Uzonyi wrote:

...

Isn't it the best time to migrate the syntax of FFI calls to Pragmas? I know it takes some time to implement the pragma support, but IIRC Lukas did that a few years ago, so dusting it off and adding support for threaded calls (which I didn't see yet) shouldn't be that hard.

Sorry, but that's a decidedly bad idea. It breaks each and every use of the FFI out there. The FFI syntax and support is what it is, so please leave it alone. If you want to implement something different, then implement something different and do it in a form that doesn't conflict with the FFI so that both can live side by side.

That's the idea: "Both formats could be supported for a while, then we could leave the current format behind and simplify the parsers/compilers, etc."

Even tools can be built to convert from one syntax to the other.

Glad to hear it. I hate the attitude that the only way to make progress is to break all code in existence. It's a lame and lazy attitude, unworthy of engineers who understand what they're doing.

Cheers, - Andreas

Levente

...

Cheers,

• Andreas

...

Both formats could be supported for a while, then we could leave the current format behind and simplify the parsers/compilers, etc.

Levente

Is there a way to throw a facade in front of FFI that can express the stuff folks want using pragmas without breaking everyone's app?

I don't know nearly as much as I'd like to about the FFI, so sorry that this question is so uninformed, but it would be cool to have our cake and eat it too, no? Just a thought...

On Mar 24, 2011, at 11:08 PM, Andreas Raab andreas.raab@gmx.de wrote:

On 3/24/2011 19:35, Levente Uzonyi wrote:

...

Isn't it the best time to migrate the syntax of FFI calls to Pragmas? I know it takes some time to implement the pragma support, but IIRC Lukas did that a few years ago, so dusting it off and adding support for threaded calls (which I didn't see yet) shouldn't be that hard.

Sorry, but that's a decidedly bad idea. It breaks each and every use of the FFI out there. The FFI syntax and support is what it is, so please leave it alone. If you want to implement something different, then implement something different and do it in a form that doesn't conflict with the FFI so that both can live side by side.

Cheers,

• Andreas

...

Both formats could be supported for a while, then we could leave the current format behind and simplify the parsers/compilers, etc.

Levente