As far as i understood , this is a way to insert own function to force VM to interrupt.
This is cool.
But i don't see another insertion point, which could make this feature complete. Suppose my plugin needs to handle some external events. Then, in my custom interrupt checker i can check if there any pending events to handle, while during interrupt i should be able to install own routine as well, which could actually handle events (like convert events to some object(s) and/or signal semaphore etc) without worrying that i'm doing this in the middle of interpreter cycle, which may damage interpreter's volatile state.
Hi Igor,
On Tue, Sep 28, 2010 at 10:48 PM, Igor Stasenko siguctua@gmail.com wrote:
As far as i understood , this is a way to insert own function to force VM to interrupt.
Alas I think it's the reverse of this. It is a way of getting the VM's heartbeat to interrupt other functions. In both Cog VMs there is a heartbeat (preferrably in a high-priroity thread) that interrupts the VM at approximately 1KHz. It does this by setting the VM's stackLimit to a value that will cause the next stack check to always fail (stacks grow down so (unsigned)-1 is the value). Then the VM checks for events as part of its stack overflow processing. This makes the stack check do double-duty, and avoids having to maintain a counter which (a) is slow due to a read-modify-write cycle and (b) runs at a variable rate because the rate at which the counter counts depends on what the VM is doing. However, the heartbeat depends on the VM performing frequent stack checks (which it does on non-primiive sends) which won't help if for example the VM is executing a long-running primitive.
The Bochs plugin executes x86 code generated by the Cog JIT when simulating the Cog VM. Bochs would happily run forever if the Cog JIT generated code that looped and I'd be hosed, except that setInterruptCheckChain allows the Bochs plugin to stop running when it is interrupted. So this isn't even close to what you want, right?
This is cool.
But i don't see another insertion point, which could make this feature complete. Suppose my plugin needs to handle some external events. Then, in my custom interrupt checker i can check if there any pending events to handle, while during interrupt i should be able to install own routine as well, which could actually handle events (like convert events to some object(s) and/or signal semaphore etc) without worrying that i'm doing this in the middle of interpreter cycle, which may damage interpreter's volatile state.
Well, the signalExternalSemaphore mechanism (at least in Cog) is thread-safe and can be used at any time form any thread to request a signal that will get performed at the next event check.
The threaded FFI I'm working on will allow one to attempt a callback at any time. The callback will simply block until it is safe for it to take over the VM. So this will also be thread-safe. But its not one yet.
HTH Eliot
P.S. I saw your other messages and will try and respond tomorrow.
-- Best regards, Igor Stasenko AKA sig.
On 29 September 2010 09:35, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi Igor,
On Tue, Sep 28, 2010 at 10:48 PM, Igor Stasenko siguctua@gmail.com wrote:
As far as i understood , this is a way to insert own function to force VM to interrupt.
Alas I think it's the reverse of this. It is a way of getting the VM's heartbeat to interrupt other functions. In both Cog VMs there is a heartbeat (preferrably in a high-priroity thread) that interrupts the VM at approximately 1KHz. It does this by setting the VM's stackLimit to a value that will cause the next stack check to always fail (stacks grow down so (unsigned)-1 is the value). Then the VM checks for events as part of its stack overflow processing. This makes the stack check do double-duty, and avoids having to maintain a counter which (a) is slow due to a read-modify-write cycle and (b) runs at a variable rate because the rate at which the counter counts depends on what the VM is doing. However, the heartbeat depends on the VM performing frequent stack checks (which it does on non-primiive sends) which won't help if for example the VM is executing a long-running primitive. The Bochs plugin executes x86 code generated by the Cog JIT when simulating the Cog VM. Bochs would happily run forever if the Cog JIT generated code that looped and I'd be hosed, except that setInterruptCheckChain allows the Bochs plugin to stop running when it is interrupted. So this isn't even close to what you want, right?
So as i understood by reverse is, that this is the way to force some other (non-main thread) to interrupt itself once heartbeat occurs. Right? Okay, this is userful as well as reverse control: Is there a thread-safe function to force VM to interrupt?
AFAIK its signalExternalSemaphore(), but what if you don't want to signal anything?
For example: i want my plugin to periodically update some small number of objects (probably smallints) held in some array.
This is cool.
But i don't see another insertion point, which could make this feature complete. Suppose my plugin needs to handle some external events. Then, in my custom interrupt checker i can check if there any pending events to handle, while during interrupt i should be able to install own routine as well, which could actually handle events (like convert events to some object(s) and/or signal semaphore etc) without worrying that i'm doing this in the middle of interpreter cycle, which may damage interpreter's volatile state.
Well, the signalExternalSemaphore mechanism (at least in Cog) is thread-safe and can be used at any time form any thread to request a signal that will get performed at the next event check.
No, its not the same. During asynchronous event, of course you are limited to semaphores. But during synchronous interrupt you can do much more.
For instance, create a linked list and append new objects directly to that list once new events available. Then language side won't even need to bother using extra primitive to pool/read events from some source , and can simply do own stuff once new object(s) 'magically' appear in such list.
See the recordMouseEvent() and its friends (sqNextEventPut, eventBuffer[1024] ...) in sqWin32Window, for better understanding my point :)
The threaded FFI I'm working on will allow one to attempt a callback at any time. The callback will simply block until it is safe for it to take over the VM. So this will also be thread-safe. But its not one yet. HTH Eliot
P.S. I saw your other messages and will try and respond tomorrow.
Good.
-- Best regards, Igor Stasenko AKA sig.
On Wed, Sep 29, 2010 at 12:20 AM, Igor Stasenko siguctua@gmail.com wrote:
On 29 September 2010 09:35, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi Igor,
On Tue, Sep 28, 2010 at 10:48 PM, Igor Stasenko siguctua@gmail.com
wrote:
As far as i understood , this is a way to insert own function to force VM to interrupt.
Alas I think it's the reverse of this. It is a way of getting the VM's
heartbeat to interrupt other functions. In both Cog VMs there is a heartbeat (preferrably in a high-priroity thread) that interrupts the VM at approximately 1KHz. It does this by setting the VM's stackLimit to a value that will cause the next stack check to always fail (stacks grow down so (unsigned)-1 is the value). Then the VM checks for events as part of its stack overflow processing. This makes the stack check do double-duty, and avoids having to maintain a counter which (a) is slow due to a read-modify-write cycle and (b) runs at a variable rate because the rate at which the counter counts depends on what the VM is doing. However, the heartbeat depends on the VM performing frequent stack checks (which it does on non-primiive sends) which won't help if for example the VM is executing a long-running primitive.
The Bochs plugin executes x86 code generated by the Cog JIT when
simulating the Cog VM. Bochs would happily run forever if the Cog JIT generated code that looped and I'd be hosed, except that setInterruptCheckChain allows the Bochs plugin to stop running when it is interrupted. So this isn't even close to what you want, right?
So as i understood by reverse is, that this is the way to force some other (non-main thread) to interrupt itself once heartbeat occurs. Right? Okay, this is userful as well as reverse control: Is there a thread-safe function to force VM to interrupt?
AFAIK its signalExternalSemaphore(), but what if you don't want to signal anything?
For example: i want my plugin to periodically update some small number of objects (probably smallints) held in some array.
This is cool.
But i don't see another insertion point, which could make this feature
complete.
Suppose my plugin needs to handle some external events. Then, in my custom interrupt checker i can check if there any pending events to handle, while during interrupt i should be able to install own routine as well, which could actually handle events (like convert events to some object(s) and/or signal semaphore
etc)
without worrying that i'm doing this in the middle of interpreter cycle, which may damage interpreter's volatile state.
Well, the signalExternalSemaphore mechanism (at least in Cog) is
thread-safe and can be used at any time form any thread to request a signal that will get performed at the next event check.
No, its not the same. During asynchronous event, of course you are limited to semaphores. But during synchronous interrupt you can do much more.
For synchronous activities one could indeed have a simple on-event-check chain. But you could do this with a process waiting on a Delay, polling occasionally, instead of calling some interrupt check function at high frequency every time the VM checks for events. It does that anyway in ioProcessEvents().
What's the use case?
For instance, create a linked list and append new objects directly to
that list once new events available. Then language side won't even need to bother using extra primitive to pool/read events from some source , and can simply do own stuff once new object(s) 'magically' appear in such list.
While one could provide a thread-safe object manipulation mechanism (I wrote a slow prototype in the VisualWorks VM in 1998) I think the best way to do this is with a callback. So Smalltalk code ends up translating from the C data into Smalltalk objects and manipulating the linked list and doing whatever else is needed. Doing otherwise really complicates the garbage collector etc. If instead one focusses on making callbacks as easy and efficient to use and possible one has a general mechanism and overall the system is simpler.
See the recordMouseEvent() and its friends (sqNextEventPut, eventBuffer[1024] ...) in sqWin32Window, for better understanding my point :)
Which makes my point well. These are simply maintaining a queue. If one has callbacks then the bulk of the Windows event handling system can be lifted up into the image. This is what Vassili Bykov did in Newspeak above my Alien callbacks. In Newspeak the Windows MainWndProc is a callback:
protected setupWindowProc = ( windowProc:: api Callback block: [:args :result | result returnInteger: (dispatchMessage: args)] stdcallArgsClass: api WindowProc )
The threaded FFI I'm working on will allow one to attempt a callback at
any time. The callback will simply block until it is safe for it to take over the VM. So this will also be thread-safe. But its not one yet.
HTH Eliot
P.S. I saw your other messages and will try and respond tomorrow.
Good.
-- Best regards, Igor Stasenko AKA sig.
-- Best regards, Igor Stasenko AKA sig.
On 9/29/2010 9:36 AM, Eliot Miranda wrote:
See the recordMouseEvent() and its friends (sqNextEventPut, eventBuffer[1024] ...) in sqWin32Window, for better understanding my point :)Which makes my point well. These are simply maintaining a queue. If one has callbacks then the bulk of the Windows event handling system can be lifted up into the image. This is what Vassili Bykov did in Newspeak above my Alien callbacks. In Newspeak the Windows MainWndProc is a callback:
Thus losing any level of platform abstraction. A bad trade-off, IMO.
Cheers, - Andreas
On Wed, Sep 29, 2010 at 10:22 AM, Andreas Raab andreas.raab@gmx.de wrote:
On 9/29/2010 9:36 AM, Eliot Miranda wrote:
See the recordMouseEvent() and its friends (sqNextEventPut, eventBuffer[1024] ...) in sqWin32Window, for better understanding my point :)
Which makes my point well. These are simply maintaining a queue. If one has callbacks then the bulk of the Windows event handling system can be lifted up into the image. This is what Vassili Bykov did in Newspeak above my Alien callbacks. In Newspeak the Windows MainWndProc is a callback:
Thus losing any level of platform abstraction. A bad trade-off, IMO.
The platform abstraction merely moves from the VM to classes in the image. But it does mean that one can properly implement Windows event semantics (things like query quit events) that the queue prevents. And it means that the VM gets simpler and complexity is handled by better facilities (in Smalltalk with its many advantages for creating and maintaining abstractions with concrete variant implementations).
But you and I have had this discussion a number of times and I take your point about the VM maintaining a black-box abstraction boundary whereas the in-image implementation is in theory porous. As I've said before this doesn't strike me as that important when Smalltalk has had boundary violators such as instVarAt:[put:] for ever and they are rarely abused. If one were to implement a native GUI event interface in Smalltalk using callbacks one would have to police the abstraction boundary. But that's easily done, and there are other benefits - the VM, being simpler, gets more longevity since changing the event interface doesn't imply a VM change - the system, having native interfaces, can support each platform's facilities in full instead of providing the lowest common denominator
So IMO it is a good trade-off.
best Eliot
Cheers,
- Andreas
Another aspect of the trade-off is releasing creativity. To change the VM you need advanced skills in both: - Smalltalk VM internals - and external libraries you want to use. That resticts dramatically the list of creators.
Plus, the difficulties for sharing your creation, you shall distribute: - either a binary version, but this means users cannot have a mixture of different features if those require a core modification, - or VMMaker patches but your audience will shrink again.
On the other hand, a VM providing a larger de facto immutable API: - somehow is more secure (a guaranty we can run very old images in very new OSes). - can maintain the illusion that despite the efforts of OS and hardware designers to make it ever more complex, a single person can still understand (almost) the whole system. - avoid the necessity to embed knowledge of dozens of different variants of different OSes in your image.
Of course, the complexity still exists under the carpet... When I contemplate all the unecessary complex knowledge from those beautiful configure/cmake scripts and macros, I'm not convinced all this cruft will be easier to modify in Smalltalk than it is in C world.
I perfectly understand too the desire of a VM maintainer to lighten the burden ;) though.
Nicolas
2010/9/29 Eliot Miranda eliot.miranda@gmail.com:
On Wed, Sep 29, 2010 at 10:22 AM, Andreas Raab andreas.raab@gmx.de wrote:
On 9/29/2010 9:36 AM, Eliot Miranda wrote:
See the recordMouseEvent() and its friends (sqNextEventPut, eventBuffer[1024] ...) in sqWin32Window, for better understanding my point :)
Which makes my point well. These are simply maintaining a queue. If one has callbacks then the bulk of the Windows event handling system can be lifted up into the image. This is what Vassili Bykov did in Newspeak above my Alien callbacks. In Newspeak the Windows MainWndProc is a callback:
Thus losing any level of platform abstraction. A bad trade-off, IMO.
The platform abstraction merely moves from the VM to classes in the image. But it does mean that one can properly implement Windows event semantics (things like query quit events) that the queue prevents. And it means that the VM gets simpler and complexity is handled by better facilities (in Smalltalk with its many advantages for creating and maintaining abstractions with concrete variant implementations). But you and I have had this discussion a number of times and I take your point about the VM maintaining a black-box abstraction boundary whereas the in-image implementation is in theory porous. As I've said before this doesn't strike me as that important when Smalltalk has had boundary violators such as instVarAt:[put:] for ever and they are rarely abused. If one were to implement a native GUI event interface in Smalltalk using callbacks one would have to police the abstraction boundary. But that's easily done, and there are other benefits
- the VM, being simpler, gets more longevity since changing the event interface doesn't imply a VM change
- the system, having native interfaces, can support each platform's facilities in full instead of providing the lowest common denominator
So IMO it is a good trade-off. best Eliot
Cheers, - Andreas
On 9/29/2010 11:53 AM, Nicolas Cellier wrote:
Another aspect of the trade-off is releasing creativity. To change the VM you need advanced skills in both:
- Smalltalk VM internals
- and external libraries you want to use.
That resticts dramatically the list of creators.
Plus, the difficulties for sharing your creation, you shall distribute:
- either a binary version, but this means users cannot have a mixture
of different features if those require a core modification,
- or VMMaker patches but your audience will shrink again.
That's why we have plugins. Keeping the core of the VM as small as possible is most definitely a Very Good Thing and not what's being discussed here. What's being discussed here is if (from that point on) one should provide layers of abstractions via plugins or if one should just expose the image directly to all of the idiosyncrasies of the platform, handing it pointers and let it make FFI calls.
It really is a matter of where you want to deal with certain aspects of the system. I prefer to deal with that level of stuff in a plugin via C because: + it provides a way to do ensure memory safety + it provides a way to provide abstractions + the code is faster to write and debug in C + access to facilities (threads, interrupts, atomicity) not available otherwise
On the downside, there is: - it's harder to modify for people who don't know C and are not set up to build a plugin - it's often aimed at the smallest common denominator (though not necessarily)
Cheers, - Andreas
On the other hand, a VM providing a larger de facto immutable API:
- somehow is more secure (a guaranty we can run very old images in
very new OSes).
- can maintain the illusion that despite the efforts of OS and
hardware designers to make it ever more complex, a single person can still understand (almost) the whole system.
- avoid the necessity to embed knowledge of dozens of different
variants of different OSes in your image.
Of course, the complexity still exists under the carpet... When I contemplate all the unecessary complex knowledge from those beautiful configure/cmake scripts and macros, I'm not convinced all this cruft will be easier to modify in Smalltalk than it is in C world.
I perfectly understand too the desire of a VM maintainer to lighten the burden ;) though.
Nicolas
2010/9/29 Eliot Mirandaeliot.miranda@gmail.com:
On Wed, Sep 29, 2010 at 10:22 AM, Andreas Raabandreas.raab@gmx.de wrote:
On 9/29/2010 9:36 AM, Eliot Miranda wrote:
See the recordMouseEvent() and its friends (sqNextEventPut, eventBuffer[1024] ...) in sqWin32Window, for better understanding my point :)Which makes my point well. These are simply maintaining a queue. If one has callbacks then the bulk of the Windows event handling system can be lifted up into the image. This is what Vassili Bykov did in Newspeak above my Alien callbacks. In Newspeak the Windows MainWndProc is a callback:
Thus losing any level of platform abstraction. A bad trade-off, IMO.
The platform abstraction merely moves from the VM to classes in the image. But it does mean that one can properly implement Windows event semantics (things like query quit events) that the queue prevents. And it means that the VM gets simpler and complexity is handled by better facilities (in Smalltalk with its many advantages for creating and maintaining abstractions with concrete variant implementations). But you and I have had this discussion a number of times and I take your point about the VM maintaining a black-box abstraction boundary whereas the in-image implementation is in theory porous. As I've said before this doesn't strike me as that important when Smalltalk has had boundary violators such as instVarAt:[put:] for ever and they are rarely abused. If one were to implement a native GUI event interface in Smalltalk using callbacks one would have to police the abstraction boundary. But that's easily done, and there are other benefits
- the VM, being simpler, gets more longevity since changing the event interface doesn't imply a VM change
- the system, having native interfaces, can support each platform's facilities in full instead of providing the lowest common denominator
So IMO it is a good trade-off. best Eliot
Cheers,
- Andreas
On 29.09.2010, at 20:53, Nicolas Cellier wrote:
When I contemplate all the unecessary complex knowledge from those beautiful configure/cmake scripts and macros, I'm not convinced all this cruft will be easier to modify in Smalltalk than it is in C world.
You still have to know C and the C libraries to do anything at that level. Using FFI does not buy you anything in this regard.
While other languages see themselves as C extensions / glue languages and happily expose C constructs directly, IMHO Squeak should not depend on FFI, but use proper VM abstractions. Supporting FFI is good, depending on it, not.
- Bert -
On Wed, Sep 29, 2010 at 12:22 PM, Andreas Raab andreas.raab@gmx.de wrote:
On 9/29/2010 11:53 AM, Nicolas Cellier wrote:
Another aspect of the trade-off is releasing creativity. To change the VM you need advanced skills in both:
- Smalltalk VM internals
- and external libraries you want to use.
That resticts dramatically the list of creators.
Plus, the difficulties for sharing your creation, you shall distribute:
- either a binary version, but this means users cannot have a mixture
of different features if those require a core modification,
- or VMMaker patches but your audience will shrink again.
That's why we have plugins. Keeping the core of the VM as small as possible is most definitely a Very Good Thing and not what's being discussed here.
I think it /is/, and you shouldn't allow your disinclination for the FFI approach blind you to the fact that one way of keeping the VM small is to produce a really strong flexible FFI that supports threading and callbacks and to implement interfaces above the line. Do you seriously believe things like the ODBC connect should be implemented in plugins?
What's being discussed here is if (from that point on) one should provide
layers of abstractions via plugins or if one should just expose the image directly to all of the idiosyncrasies of the platform, handing it pointers and let it make FFI calls.
In the case of event callbacks its a different issue. The current event queue abstraction is broken in that it prevents Smalltalk code from answering questions asked through events such as WM_QUERYENDSESSION because the VM is forced to queue events, and hence can only provide a default answer in the window proc. So there are areas where having proper callbacks is required to be able to integrate properly with the host operating system.
I don't disagree with what you say about plugins, or about maintaining cross-platform abstractions in the VM for core functionality that is common across platforms, but I think its clear one needs true callbacks, and that at least for integrating with the GUI this is the best way to implement it.
It really is a matter of where you want to deal with certain aspects of the
system. I prefer to deal with that level of stuff in a plugin via C because:
- it provides a way to do ensure memory safety
there's nothing in plugins per-se to maintain memory safety. A mistake in a plugin is as destructive as a mistake in an FFI call. In either case a wrapper around a particular call can validate and provide safety.
- it provides a way to provide abstractions
- so does Smalltalk, and t least in SMalltalk we have inheritance to enable us to express cross-platform APIs in abstract classes and map these down onto particular platforms in concrete subclasses. This is /far better/ than the sprawling mess that is the platforms tree in the VM.
- the code is faster to write and debug in C
this is debatable. In the VW FFI with the ability to catch exceptions and pass them back up (doesn't always work, a bad emory corruption may crash the entire system) makes thing easier. But low-level debugging is painful in general.
- access to facilities (threads, interrupts, atomicity) not available
otherwise
Again, a strong FFI provides at least some of these facilities. Not everything can be done through the FFI, but a lot can be done, and elegantly and extensibly, by Smalltalk programmers, not a few VM specialists.
On the downside, there is:
- it's harder to modify for people who don't know C and are not set up to
build a plugin
- it's often aimed at the smallest common denominator (though not
necessarily)
- it is a far more fixed interface that is far harder to evolve
best, Eliot
Cheers,
- Andreas
On the other hand, a VM providing a larger de facto immutable API:
- somehow is more secure (a guaranty we can run very old images in
very new OSes).
- can maintain the illusion that despite the efforts of OS and
hardware designers to make it ever more complex, a single person can still understand (almost) the whole system.
- avoid the necessity to embed knowledge of dozens of different
variants of different OSes in your image.
Of course, the complexity still exists under the carpet... When I contemplate all the unecessary complex knowledge from those beautiful configure/cmake scripts and macros, I'm not convinced all this cruft will be easier to modify in Smalltalk than it is in C world.
I perfectly understand too the desire of a VM maintainer to lighten the burden ;) though.
Nicolas
2010/9/29 Eliot Mirandaeliot.miranda@gmail.com:
On Wed, Sep 29, 2010 at 10:22 AM, Andreas Raabandreas.raab@gmx.de wrote:
On 9/29/2010 9:36 AM, Eliot Miranda wrote:
See the recordMouseEvent() and its friends (sqNextEventPut, eventBuffer[1024] ...) in sqWin32Window, for better understanding my point :)
Which makes my point well. These are simply maintaining a queue. If one has callbacks then the bulk of the Windows event handling system can be lifted up into the image. This is what Vassili Bykov did in Newspeak above my Alien callbacks. In Newspeak the Windows MainWndProc is a callback:
Thus losing any level of platform abstraction. A bad trade-off, IMO.
The platform abstraction merely moves from the VM to classes in the image. But it does mean that one can properly implement Windows event semantics (things like query quit events) that the queue prevents. And it means that the VM gets simpler and complexity is handled by better facilities (in Smalltalk with its many advantages for creating and maintaining abstractions with concrete variant implementations). But you and I have had this discussion a number of times and I take your point about the VM maintaining a black-box abstraction boundary whereas the in-image implementation is in theory porous. As I've said before this doesn't strike me as that important when Smalltalk has had boundary violators such as instVarAt:[put:] for ever and they are rarely abused. If one were to implement a native GUI event interface in Smalltalk using callbacks one would have to police the abstraction boundary. But that's easily done, and there are other benefits
- the VM, being simpler, gets more longevity since changing the event
interface doesn't imply a VM change
- the system, having native interfaces, can support each platform's
facilities in full instead of providing the lowest common denominator So IMO it is a good trade-off. best Eliot
Cheers,
- Andreas
On 29 September 2010 19:36, Eliot Miranda eliot.miranda@gmail.com wrote:
On Wed, Sep 29, 2010 at 12:20 AM, Igor Stasenko siguctua@gmail.com wrote:
On 29 September 2010 09:35, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi Igor,
On Tue, Sep 28, 2010 at 10:48 PM, Igor Stasenko siguctua@gmail.com wrote:
As far as i understood , this is a way to insert own function to force VM to interrupt.
Alas I think it's the reverse of this. It is a way of getting the VM's heartbeat to interrupt other functions. In both Cog VMs there is a heartbeat (preferrably in a high-priroity thread) that interrupts the VM at approximately 1KHz. It does this by setting the VM's stackLimit to a value that will cause the next stack check to always fail (stacks grow down so (unsigned)-1 is the value). Then the VM checks for events as part of its stack overflow processing. This makes the stack check do double-duty, and avoids having to maintain a counter which (a) is slow due to a read-modify-write cycle and (b) runs at a variable rate because the rate at which the counter counts depends on what the VM is doing. However, the heartbeat depends on the VM performing frequent stack checks (which it does on non-primiive sends) which won't help if for example the VM is executing a long-running primitive. The Bochs plugin executes x86 code generated by the Cog JIT when simulating the Cog VM. Bochs would happily run forever if the Cog JIT generated code that looped and I'd be hosed, except that setInterruptCheckChain allows the Bochs plugin to stop running when it is interrupted. So this isn't even close to what you want, right?
So as i understood by reverse is, that this is the way to force some other (non-main thread) to interrupt itself once heartbeat occurs. Right? Okay, this is userful as well as reverse control: Is there a thread-safe function to force VM to interrupt?
AFAIK its signalExternalSemaphore(), but what if you don't want to signal anything?
For example: i want my plugin to periodically update some small number of objects (probably smallints) held in some array.
This is cool.
But i don't see another insertion point, which could make this feature complete. Suppose my plugin needs to handle some external events. Then, in my custom interrupt checker i can check if there any pending events to handle, while during interrupt i should be able to install own routine as well, which could actually handle events (like convert events to some object(s) and/or signal semaphore etc) without worrying that i'm doing this in the middle of interpreter cycle, which may damage interpreter's volatile state.
Well, the signalExternalSemaphore mechanism (at least in Cog) is thread-safe and can be used at any time form any thread to request a signal that will get performed at the next event check.
No, its not the same. During asynchronous event, of course you are limited to semaphores. But during synchronous interrupt you can do much more.
For synchronous activities one could indeed have a simple on-event-check chain. But you could do this with a process waiting on a Delay, polling occasionally, instead of calling some interrupt check function at high frequency every time the VM checks for events. It does that anyway in ioProcessEvents(). What's the use case?
For instance, create a linked list and append new objects directly to that list once new events available. Then language side won't even need to bother using extra primitive to pool/read events from some source , and can simply do own stuff once new object(s) 'magically' appear in such list.
While one could provide a thread-safe object manipulation mechanism (I wrote a slow prototype in the VisualWorks VM in 1998) I think the best way to do this is with a callback. So Smalltalk code ends up translating from the C data into Smalltalk objects and manipulating the linked list and doing whatever else is needed. Doing otherwise really complicates the garbage collector etc. If instead one focusses on making callbacks as easy and efficient to use and possible one has a general mechanism and overall the system is simpler.
See the recordMouseEvent() and its friends (sqNextEventPut, eventBuffer[1024] ...) in sqWin32Window, for better understanding my point :)
Which makes my point well. These are simply maintaining a queue. If one has callbacks then the bulk of the Windows event handling system can be lifted up into the image. This is what Vassili Bykov did in Newspeak above my Alien callbacks. In Newspeak the Windows MainWndProc is a callback: protected setupWindowProc = ( windowProc:: api Callback block: [:args :result | result returnInteger: (dispatchMessage: args)] stdcallArgsClass: api WindowProc )
Yep.. but for that, you should be able to give me a choice: - how i can disable a default VM event handling , so then i can install and use callbacks?
There are multiple things right now, which won't allow me to do that. Not reentrant interpret(), no way to control why and when VM deciding to interrupt and call ioHandleEvents(). And now way to install own ioHandleEvents(), in order to replace a default one.
My own idea was to have a function pointer, instead direct call to ioProcessEvents(), and let plugins to install/replace this pointer with anything we want. This is similar to chaining, you made for hearbeat thread.
The threaded FFI I'm working on will allow one to attempt a callback at any time. The callback will simply block until it is safe for it to take over the VM. So this will also be thread-safe. But its not one yet. HTH Eliot
P.S. I saw your other messages and will try and respond tomorrow.
Good.
-- Best regards, Igor Stasenko AKA sig.
-- Best regards, Igor Stasenko AKA sig.
On 29 September 2010 22:40, Bert Freudenberg bert@freudenbergs.de wrote:
On 29.09.2010, at 20:53, Nicolas Cellier wrote:
When I contemplate all the unecessary complex knowledge from those beautiful configure/cmake scripts and macros, I'm not convinced all this cruft will be easier to modify in Smalltalk than it is in C world.
You still have to know C and the C libraries to do anything at that level. Using FFI does not buy you anything in this regard.
While other languages see themselves as C extensions / glue languages and happily expose C constructs directly, IMHO Squeak should not depend on FFI, but use proper VM abstractions. Supporting FFI is good, depending on it, not.
My intent is to make VM flexible. Sure thing, VM should provide a default OS-specific layer. The pity is, that i can't alter/replace it, when using FFI.
- Bert -
On 29 September 2010 23:03, Eliot Miranda eliot.miranda@gmail.com wrote:
On Wed, Sep 29, 2010 at 12:22 PM, Andreas Raab andreas.raab@gmx.de wrote:
On 9/29/2010 11:53 AM, Nicolas Cellier wrote:
Another aspect of the trade-off is releasing creativity. To change the VM you need advanced skills in both:
- Smalltalk VM internals
- and external libraries you want to use.
That resticts dramatically the list of creators.
Plus, the difficulties for sharing your creation, you shall distribute:
- either a binary version, but this means users cannot have a mixture
of different features if those require a core modification,
- or VMMaker patches but your audience will shrink again.
That's why we have plugins. Keeping the core of the VM as small as possible is most definitely a Very Good Thing and not what's being discussed here.
I think it /is/, and you shouldn't allow your disinclination for the FFI approach blind you to the fact that one way of keeping the VM small is to produce a really strong flexible FFI that supports threading and callbacks and to implement interfaces above the line. Do you seriously believe things like the ODBC connect should be implemented in plugins?
What's being discussed here is if (from that point on) one should provide layers of abstractions via plugins or if one should just expose the image directly to all of the idiosyncrasies of the platform, handing it pointers and let it make FFI calls.
In the case of event callbacks its a different issue. The current event queue abstraction is broken in that it prevents Smalltalk code from answering questions asked through events such as WM_QUERYENDSESSION because the VM is forced to queue events, and hence can only provide a default answer in the window proc. So there are areas where having proper callbacks is required to be able to integrate properly with the host operating system. I don't disagree with what you say about plugins, or about maintaining cross-platform abstractions in the VM for core functionality that is common across platforms, but I think its clear one needs true callbacks, and that at least for integrating with the GUI this is the best way to implement it.
It really is a matter of where you want to deal with certain aspects of the system. I prefer to deal with that level of stuff in a plugin via C because:
- it provides a way to do ensure memory safety
there's nothing in plugins per-se to maintain memory safety. A mistake in a plugin is as destructive as a mistake in an FFI call. In either case a wrapper around a particular call can validate and provide safety.
- it provides a way to provide abstractions
- so does Smalltalk, and t least in SMalltalk we have inheritance to enable us to express cross-platform APIs in abstract classes and map these down onto particular platforms in concrete subclasses. This is /far better/ than the sprawling mess that is the platforms tree in the VM.
- the code is faster to write and debug in C
this is debatable. In the VW FFI with the ability to catch exceptions and pass them back up (doesn't always work, a bad emory corruption may crash the entire system) makes thing easier. But low-level debugging is painful in general.
- access to facilities (threads, interrupts, atomicity) not available otherwise
Again, a strong FFI provides at least some of these facilities. Not everything can be done through the FFI, but a lot can be done, and elegantly and extensibly, by Smalltalk programmers, not a few VM specialists.
On the downside, there is:
- it's harder to modify for people who don't know C and are not set up to build a plugin
- it's often aimed at the smallest common denominator (though not necessarily)
- it is a far more fixed interface that is far harder to evolve
Eliot, i don't know what to say in addition. I have same thoughts as you.
Its not a question, that VM should provide a default platform abstraction layer(s). But the question, how i can make part's of it optional, or easily alter/replace them with own implementation (either plugin of ffi) in case of need, without the need of shipping of own, custom built VM?
Where ends a VM's mandatory functionality and starts an optional one? a) providing UI & event handling? Apparently not for those who wants to run VM as GUI-less command-line tool or implement own GUI framework. b) providing a network layer? Apparently not for those, who not using network in their application(s), and instead want to make sure that no network activity possible. Or again, wants to provide own network layer.
FFI is not a silver bullet. But my point is not putting you before choice: use C or FFI. Its about making VM more flexible at run-time, not at compile time.
best, Eliot
it is a far more fixed interface that is far harder to evolve
...
Its about making VM more flexible at run-time, not at compile time.
These are exactly my recent thoughts and which led me to get reacquainted with Forth as the basis for a "re-programmable" plug-in. It's hard then not to consider a Forth based VM bootstrapped from the image itself, although I'm not yet sure this is even a sane idea :-) I wouldn't be surprised if someone has already considered and rejected it.
-D
On 9/29/2010 1:03 PM, Eliot Miranda wrote:
I think it /is/, and you shouldn't allow your disinclination for the FFI approach blind you to the fact that one way of keeping the VM small is to produce a really strong flexible FFI that supports threading and callbacks and to implement interfaces above the line. Do you seriously believe things like the ODBC connect should be implemented in plugins?
Absolutely. Memory safety being the prime reason.
In the case of event callbacks its a different issue. The current event queue abstraction is broken in that it prevents Smalltalk code from answering questions asked through events such as WM_QUERYENDSESSION because the VM is forced to queue events, and hence can only provide a default answer in the window proc. So there are areas where having proper callbacks is required to be able to integrate properly with the host operating system.
Please. The issue here is that our event mechanism is a bit simplistic but the Android VM is a great example of how to deal with these issues properly. Sure, callbacks could address this too, but claiming *only* callbacks can address this is simply wrong.
I don't disagree with what you say about plugins, or about maintaining cross-platform abstractions in the VM for core functionality that is common across platforms, but I think its clear one needs true callbacks, and that at least for integrating with the GUI this is the best way to implement it.
A solution that inherently implies exposure of raw pointers to users isn't what I would call the "best" solution. That said, I'm not arguing against callbacks - when you need them you need them. And there are practical situations where it's incredibly handy to have them. And yet, would I design the system so that it depends at its core on such facilities? Hell, no, because in addition to being memory-unsafe, porting an FFI is *much* harder than porting a bit of C support code.
It really is a matter of where you want to deal with certain aspects of the system. I prefer to deal with that level of stuff in a plugin via C because: + it provides a way to do ensure memory safetythere's nothing in plugins per-se to maintain memory safety. A mistake in a plugin is as destructive as a mistake in an FFI call. In either case a wrapper around a particular call can validate and provide safety.
In theory, yes. In practice such wrappers *do* exist in plugins and they do *not* exist in FFI calls. Why? Good question. Probably because people understand that in C land there are more constraints and they are mentally "set up" to keep an eye on the constraints and to make sure the incoming stuff is okay.
+ it provides a way to provide abstractions
- so does Smalltalk, and t least in SMalltalk we have inheritance to
enable us to express cross-platform APIs in abstract classes and map these down onto particular platforms in concrete subclasses. This is /far better/ than the sprawling mess that is the platforms tree in the VM.
In theory yes. In practice, who knows. Having looked at some Smalltalks that expose all the platform details (like Dolphin) it seems to me that the equation of "platform FFI code * number of platforms" will be /far more/ of a sprawling mess. Which is probably why (unless I'm mistaken in which case I'll appreciate a correction) precisely none of the cross-platform Smalltalks do that.
+ the code is faster to write and debug in Cthis is debatable.
Hardly. Let's start with "#include <windows.h>" (or whatever platform stuff you happen to need). You're losing right there. The problem is that the system isn't set up to seamlessly interact with C. Even if you've got some auto-magic header file importer that creates structs and defines and stuff, as long as it takes more than, i.e.,
winMessageBox: contents title: title <include: 'windows.h'> MessageBox(NULL, contents, title, MB_OK + MB_ICONWARNING).
you're going to lose because it takes so much longer to get to the point of actually writing the code you're trying to write.
Secondly, you *really* need to try the latest Visual Studio and play with its debugger - I was blown away while debugging SqueakSSL when I hit a typical NULL pointer, fixed the code in my plugin DLL, continued in VS and the underlying Squeak KEPT RUNNING! Holy crap. This stuff has come a looooong way.
In the VW FFI with the ability to catch exceptions and pass them back up (doesn't always work, a bad emory corruption may crash the entire system) makes thing easier. But low-level debugging is painful in general.
+ access to facilities (threads, interrupts, atomicity) not available otherwiseAgain, a strong FFI provides at least some of these facilities. Not everything can be done through the FFI, but a lot can be done, and elegantly and extensibly, by Smalltalk programmers, not a few VM specialists.
I call BS. The reality is that one needs to know three kinds of things, only one of which would be helped by using the FFI:
1) The "mechanics" of writing a plugin, i.e., slang and compilation. This is the part where the FFI would help, but so would better APIs for marshaling. While it's "cool" that one can run slang plugins in Squeak it is also completely and utterly *useless* when it comes to integrating platform specific stuff and having to deal with Slang, VMMaker and the awful build setups, and then -on top of that- the actual work you're trying to do is just a bit too much.
2) Understanding the rules of interfacing the external world, i.e., when can pointers be kept, what does GC do to your objects etc. None of this is helped by using the FFI.
3) Understanding the actual domain code. Again, none of this is helped by the FFI (to the contrary because nobody will find examples for the weird Smalltalk selectors we use on the web and people will in turn not know where to find the actual value for MB_OK or somesuch).
Basically, the assumption that *all* the difficulty is in writing the marshaling code is nonsense. There is some of it, true, but that is entirely our fault for providing poor integration APIs.
On the downside, there is: - it's harder to modify for people who don't know C and are not set up to build a plugin - it's often aimed at the smallest common denominator (though not necessarily)
- it is a far more fixed interface that is far harder to evolve
Yes, but when you call it "stable" instead of "fixed" and "robust" instead of "harder to evolve" it becomes a plus :-)
Cheers, - Andreas
Cheers, - Andreas On the other hand, a VM providing a larger de facto immutable API: - somehow is more secure (a guaranty we can run very old images in very new OSes). - can maintain the illusion that despite the efforts of OS and hardware designers to make it ever more complex, a single person can still understand (almost) the whole system. - avoid the necessity to embed knowledge of dozens of different variants of different OSes in your image. Of course, the complexity still exists under the carpet... When I contemplate all the unecessary complex knowledge from those beautiful configure/cmake scripts and macros, I'm not convinced all this cruft will be easier to modify in Smalltalk than it is in C world. I perfectly understand too the desire of a VM maintainer to lighten the burden ;) though. Nicolas 2010/9/29 Eliot Miranda<eliot.miranda@gmail.com <mailto:eliot.miranda@gmail.com>>: On Wed, Sep 29, 2010 at 10:22 AM, Andreas Raab<andreas.raab@gmx.de <mailto:andreas.raab@gmx.de>> wrote: On 9/29/2010 9:36 AM, Eliot Miranda wrote: See the recordMouseEvent() and its friends (sqNextEventPut, eventBuffer[1024] ...) in sqWin32Window, for better understanding my point :) Which makes my point well. These are simply maintaining a queue. If one has callbacks then the bulk of the Windows event handling system can be lifted up into the image. This is what Vassili Bykov did in Newspeak above my Alien callbacks. In Newspeak the Windows MainWndProc is a callback: Thus losing any level of platform abstraction. A bad trade-off, IMO. The platform abstraction merely moves from the VM to classes in the image. But it does mean that one can properly implement Windows event semantics (things like query quit events) that the queue prevents. And it means that the VM gets simpler and complexity is handled by better facilities (in Smalltalk with its many advantages for creating and maintaining abstractions with concrete variant implementations). But you and I have had this discussion a number of times and I take your point about the VM maintaining a black-box abstraction boundary whereas the in-image implementation is in theory porous. As I've said before this doesn't strike me as that important when Smalltalk has had boundary violators such as instVarAt:[put:] for ever and they are rarely abused. If one were to implement a native GUI event interface in Smalltalk using callbacks one would have to police the abstraction boundary. But that's easily done, and there are other benefits - the VM, being simpler, gets more longevity since changing the event interface doesn't imply a VM change - the system, having native interfaces, can support each platform's facilities in full instead of providing the lowest common denominator So IMO it is a good trade-off. best Eliot Cheers, - Andreas
On 30 September 2010 02:22, Andreas Raab andreas.raab@gmx.de wrote:
On 9/29/2010 1:03 PM, Eliot Miranda wrote:
I think it /is/, and you shouldn't allow your disinclination for the FFI approach blind you to the fact that one way of keeping the VM small is to produce a really strong flexible FFI that supports threading and callbacks and to implement interfaces above the line. Do you seriously believe things like the ODBC connect should be implemented in plugins?
Absolutely. Memory safety being the prime reason.
In the case of event callbacks its a different issue. The current event queue abstraction is broken in that it prevents Smalltalk code from answering questions asked through events such as WM_QUERYENDSESSION because the VM is forced to queue events, and hence can only provide a default answer in the window proc. So there are areas where having proper callbacks is required to be able to integrate properly with the host operating system.
Please. The issue here is that our event mechanism is a bit simplistic but the Android VM is a great example of how to deal with these issues properly. Sure, callbacks could address this too, but claiming *only* callbacks can address this is simply wrong.
I don't disagree with what you say about plugins, or about maintaining cross-platform abstractions in the VM for core functionality that is common across platforms, but I think its clear one needs true callbacks, and that at least for integrating with the GUI this is the best way to implement it.
A solution that inherently implies exposure of raw pointers to users isn't what I would call the "best" solution. That said, I'm not arguing against callbacks - when you need them you need them. And there are practical situations where it's incredibly handy to have them. And yet, would I design the system so that it depends at its core on such facilities? Hell, no, because in addition to being memory-unsafe, porting an FFI is *much* harder than porting a bit of C support code.
It really is a matter of where you want to deal with certain aspects of the system. I prefer to deal with that level of stuff in a plugin via C because: + it provides a way to do ensure memory safety
there's nothing in plugins per-se to maintain memory safety. A mistake in a plugin is as destructive as a mistake in an FFI call. In either case a wrapper around a particular call can validate and provide safety.
In theory, yes. In practice such wrappers *do* exist in plugins and they do *not* exist in FFI calls. Why? Good question. Probably because people understand that in C land there are more constraints and they are mentally "set up" to keep an eye on the constraints and to make sure the incoming stuff is okay.
+ it provides a way to provide abstractions
- so does Smalltalk, and t least in SMalltalk we have inheritance to
enable us to express cross-platform APIs in abstract classes and map these down onto particular platforms in concrete subclasses. This is /far better/ than the sprawling mess that is the platforms tree in the VM.
In theory yes. In practice, who knows. Having looked at some Smalltalks that expose all the platform details (like Dolphin) it seems to me that the equation of "platform FFI code * number of platforms" will be /far more/ of a sprawling mess. Which is probably why (unless I'm mistaken in which case I'll appreciate a correction) precisely none of the cross-platform Smalltalks do that.
+ the code is faster to write and debug in C
this is debatable.
Hardly. Let's start with "#include <windows.h>" (or whatever platform stuff you happen to need). You're losing right there. The problem is that the system isn't set up to seamlessly interact with C. Even if you've got some auto-magic header file importer that creates structs and defines and stuff, as long as it takes more than, i.e.,
winMessageBox: contents title: title <include: 'windows.h'> MessageBox(NULL, contents, title, MB_OK + MB_ICONWARNING).
you're going to lose because it takes so much longer to get to the point of actually writing the code you're trying to write.
Secondly, you *really* need to try the latest Visual Studio and play with its debugger - I was blown away while debugging SqueakSSL when I hit a typical NULL pointer, fixed the code in my plugin DLL, continued in VS and the underlying Squeak KEPT RUNNING! Holy crap. This stuff has come a looooong way.
In the VW FFI with the ability to catch exceptions and pass them back up (doesn't always work, a bad emory corruption may crash the entire system) makes thing easier. But low-level debugging is painful in general.
+ access to facilities (threads, interrupts, atomicity) not available otherwise
Again, a strong FFI provides at least some of these facilities. Not everything can be done through the FFI, but a lot can be done, and elegantly and extensibly, by Smalltalk programmers, not a few VM specialists.
I call BS. The reality is that one needs to know three kinds of things, only one of which would be helped by using the FFI:
- The "mechanics" of writing a plugin, i.e., slang and compilation. This is
the part where the FFI would help, but so would better APIs for marshaling. While it's "cool" that one can run slang plugins in Squeak it is also completely and utterly *useless* when it comes to integrating platform specific stuff and having to deal with Slang, VMMaker and the awful build setups, and then -on top of that- the actual work you're trying to do is just a bit too much.
And i am here looking for a way to make it less painful. If one can make own stuff to work with OS via FFI, why he needs to go into slang , VMMaker, cmake, gcc jungles? Probably, when he consider that FFI is good, but have to use more safer solution, then he free to go and implement plugin or change VM, whatever. But for prototyping and fast implementation, FFI much better.
- Understanding the rules of interfacing the external world, i.e., when can
pointers be kept, what does GC do to your objects etc. None of this is helped by using the FFI.
- Understanding the actual domain code. Again, none of this is helped by
the FFI (to the contrary because nobody will find examples for the weird Smalltalk selectors we use on the web and people will in turn not know where to find the actual value for MB_OK or somesuch).
Basically, the assumption that *all* the difficulty is in writing the marshaling code is nonsense. There is some of it, true, but that is entirely our fault for providing poor integration APIs.
On the downside, there is: - it's harder to modify for people who don't know C and are not set up to build a plugin - it's often aimed at the smallest common denominator (though not necessarily)
- it is a far more fixed interface that is far harder to evolve
Yes, but when you call it "stable" instead of "fixed" and "robust" instead of "harder to evolve" it becomes a plus :-)
Not in context of current discussion. VM has to evolve and improve over a time. And we here to discuss what best path to take to improve VM, not 'lets freeze it for ages'. If you want to see something stable - feel free to browse the SqueakMap and SqueakSource, there you'll find many 'stable' stuff, which done once and then left unattended and not working under current VMs and images.
<sarcasm> Why Croquet using FFI for OpenGL binding, why not writing a plugin? It would be much more 'stable' and 'robust' and secure. </sarcasm>
Cheers, - Andreas
Cheers, - Andreas
On the other hand, a VM providing a larger de facto immutable API: - somehow is more secure (a guaranty we can run very old images in very new OSes). - can maintain the illusion that despite the efforts of OS and hardware designers to make it ever more complex, a single person can still understand (almost) the whole system. - avoid the necessity to embed knowledge of dozens of different variants of different OSes in your image.
Of course, the complexity still exists under the carpet... When I contemplate all the unecessary complex knowledge from those beautiful configure/cmake scripts and macros, I'm not convinced all this cruft will be easier to modify in Smalltalk than it is in C world.
I perfectly understand too the desire of a VM maintainer to lighten the burden ;) though.
Nicolas
2010/9/29 Eliot Miranda<eliot.miranda@gmail.com mailto:eliot.miranda@gmail.com>:
On Wed, Sep 29, 2010 at 10:22 AM, Andreas Raab<andreas.raab@gmx.de mailto:andreas.raab@gmx.de> wrote:
On 9/29/2010 9:36 AM, Eliot Miranda wrote:
See the recordMouseEvent() and its friends (sqNextEventPut, eventBuffer[1024] ...) in sqWin32Window, for better understanding my point :)
Which makes my point well. These are simply maintaining a queue. If one has callbacks then the bulk of the Windows event handling system can be lifted up into the image. This is what Vassili Bykov did in Newspeak above my Alien callbacks. In Newspeak the Windows MainWndProc is a callback:
Thus losing any level of platform abstraction. A bad trade-off, IMO.
The platform abstraction merely moves from the VM to classes in the image. But it does mean that one can properly implement Windows event semantics (things like query quit events) that the queue prevents. And it means that the VM gets simpler and complexity is handled by better facilities (in Smalltalk with its many advantages for creating and maintaining abstractions with concrete variant implementations). But you and I have had this discussion a number of times and I take your point about the VM maintaining a black-box abstraction boundary whereas the in-image implementation is in theory porous. As I've said before this doesn't strike me as that important when Smalltalk has had boundary violators such as instVarAt:[put:] for ever and they are rarely abused. If one were to implement a native GUI event interface in Smalltalk using callbacks one would have to police the abstraction boundary. But that's easily done, and there are other benefits - the VM, being simpler, gets more longevity since changing the event interface doesn't imply a VM change - the system, having native interfaces, can support each platform's facilities in full instead of providing the lowest common denominator So IMO it is a good trade-off. best Eliot
Cheers, - Andreas
[Side note: Please don't quote entire posts if you're going to reply 2/3rds of the way down. I had to scroll three times through this post to even find what you replied to]
On 9/29/2010 5:50 PM, Igor Stasenko wrote:
- The "mechanics" of writing a plugin, i.e., slang and compilation. This is
the part where the FFI would help, but so would better APIs for marshaling. While it's "cool" that one can run slang plugins in Squeak it is also completely and utterly *useless* when it comes to integrating platform specific stuff and having to deal with Slang, VMMaker and the awful build setups, and then -on top of that- the actual work you're trying to do is just a bit too much.
And i am here looking for a way to make it less painful. If one can make own stuff to work with OS via FFI, why he needs to go into slang , VMMaker, cmake, gcc jungles?
One word: Memory safety. Oh, that's two :-)
Probably, when he consider that FFI is good, but have to use more safer solution, then he free to go and implement plugin or change VM, whatever. But for prototyping and fast implementation, FFI much better.
See, in my experience that claim is simply wrong. How can you say that (for example) a little test program like the one that I just wrote to experiment with a Windows function would be faster to write using the FFI?
------------------------------ CredTest.cpp ----------------------------
// CredTest.cpp : Defines the entry point for the console application. //
#include "stdafx.h" #include <Windows.h> #include <wincred.h>
LPTSTR readCredential(LPTSTR targetName) { CREDENTIAL *cred;
if(CredRead(targetName, CRED_TYPE_GENERIC, 0, &cred)) { LPTSTR result = (LPTSTR) calloc(cred->CredentialBlobSize+1, sizeof(TCHAR)); memcpy(result, cred->CredentialBlob, cred->CredentialBlobSize); return result; } printf("CredRead failed: %d\n", GetLastError()); return NULL; }
int _tmain(int argc, _TCHAR* argv[]) { LPTSTR credential = readCredential(L"TERMSRV/devmsapps01"); printf("Credential: %s\n", credential); return 0; } ----------------------------------------------------------------------
You can imagine that it took me some ten minutes or so (incl. firing up Visual Studio, creating a new project etc) to write this. Now do that in Squeak and keep in mind how much of your time you're wasting in the overhead of declaring the types, functions, defines. That's why I'm saying that unless you can get the equivalent of #include <windows.h> you're not even in the same ballpark.
<sarcasm> Why Croquet using FFI for OpenGL binding, why not writing a plugin? It would be much more 'stable' and 'robust' and secure. </sarcasm>
Only one reason: The poor integration APIs. I couldn't figure out how to generate proper plugin glue (and later I had lost the source for generating this stufF). And we *paid* for it. Weeks and weeks of obscure crash reports that we couldn't figure out until *finally* we realized that when using client state operations a GC may occur if the rendering is interrupted "just so". We fixed this by (guess what) moving these operations into a plugin.
Cheers, - Andreas
On 30 September 2010 04:30, Andreas Raab andreas.raab@gmx.de wrote:
[Side note: Please don't quote entire posts if you're going to reply 2/3rds of the way down. I had to scroll three times through this post to even find what you replied to]
On 9/29/2010 5:50 PM, Igor Stasenko wrote:
- The "mechanics" of writing a plugin, i.e., slang and compilation. This
is the part where the FFI would help, but so would better APIs for marshaling. While it's "cool" that one can run slang plugins in Squeak it is also completely and utterly *useless* when it comes to integrating platform specific stuff and having to deal with Slang, VMMaker and the awful build setups, and then -on top of that- the actual work you're trying to do is just a bit too much.
And i am here looking for a way to make it less painful. If one can make own stuff to work with OS via FFI, why he needs to go into slang , VMMaker, cmake, gcc jungles?
One word: Memory safety. Oh, that's two :-)
Probably, when he consider that FFI is good, but have to use more safer solution, then he free to go and implement plugin or change VM, whatever. But for prototyping and fast implementation, FFI much better.
See, in my experience that claim is simply wrong. How can you say that (for example) a little test program like the one that I just wrote to experiment with a Windows function would be faster to write using the FFI?
------------------------------ CredTest.cpp ----------------------------
// CredTest.cpp : Defines the entry point for the console application. //
#include "stdafx.h" #include <Windows.h> #include <wincred.h>
LPTSTR readCredential(LPTSTR targetName) { CREDENTIAL *cred;
if(CredRead(targetName, CRED_TYPE_GENERIC, 0, &cred)) { LPTSTR result = (LPTSTR) calloc(cred->CredentialBlobSize+1, sizeof(TCHAR)); memcpy(result, cred->CredentialBlob, cred->CredentialBlobSize); return result; } printf("CredRead failed: %d\n", GetLastError()); return NULL; }
int _tmain(int argc, _TCHAR* argv[]) { LPTSTR credential = readCredential(L"TERMSRV/devmsapps01"); printf("Credential: %s\n", credential); return 0; }
You can imagine that it took me some ten minutes or so (incl. firing up Visual Studio, creating a new project etc) to write this. Now do that in Squeak and keep in mind how much of your time you're wasting in the overhead of declaring the types, functions, defines. That's why I'm saying that unless you can get the equivalent of #include <windows.h> you're not even in the same ballpark.
Wait, in the above you forgot to show me the glue code which exposing this function via some primitive. And you forgot to write the list of steps (in addition to opening Visual studio), how you rebuilding VM using VMMaker , compiling and linking, and finally deploying VM with new plugin, and then telling others that they can download new version of VM, so they could try your new stuff. Now i doubt if you do that, it will take ten minutes or so.
<sarcasm> Why Croquet using FFI for OpenGL binding, why not writing a plugin? It would be much more 'stable' and 'robust' and secure. </sarcasm>
Only one reason: The poor integration APIs. I couldn't figure out how to generate proper plugin glue (and later I had lost the source for generating this stufF). And we *paid* for it. Weeks and weeks of obscure crash reports that we couldn't figure out until *finally* we realized that when using client state operations a GC may occur if the rendering is interrupted "just so". We fixed this by (guess what) moving these operations into a plugin.
No objections here. Shit happens. And its really don't matters where: either in language/FFI or in C. You still have to fix that.
Cheers, - Andreas
On 9/29/2010 6:48 PM, Igor Stasenko wrote:
On 30 September 2010 04:30, Andreas Raabandreas.raab@gmx.de wrote:
See, in my experience that claim is simply wrong. How can you say that (for example) a little test program like the one that I just wrote to experiment with a Windows function would be faster to write using the FFI?
------------------------------ CredTest.cpp ----------------------------
// CredTest.cpp : Defines the entry point for the console application. //
#include "stdafx.h" #include<Windows.h> #include<wincred.h>
LPTSTR readCredential(LPTSTR targetName) { CREDENTIAL *cred;
if(CredRead(targetName, CRED_TYPE_GENERIC, 0,&cred)) { LPTSTR result = (LPTSTR) calloc(cred->CredentialBlobSize+1,sizeof(TCHAR)); memcpy(result, cred->CredentialBlob, cred->CredentialBlobSize); return result; } printf("CredRead failed: %d\n", GetLastError()); return NULL; }
int _tmain(int argc, _TCHAR* argv[]) { LPTSTR credential = readCredential(L"TERMSRV/devmsapps01"); printf("Credential: %s\n", credential); return 0; }
You can imagine that it took me some ten minutes or so (incl. firing up Visual Studio, creating a new project etc) to write this. Now do that in Squeak and keep in mind how much of your time you're wasting in the overhead of declaring the types, functions, defines. That's why I'm saying that unless you can get the equivalent of #include<windows.h> you're not even in the same ballpark.
Wait, in the above you forgot to show me the glue code which exposing this function via some primitive. And you forgot to write the list of steps (in addition to opening Visual studio), how you rebuilding VM using VMMaker , compiling and linking, and finally deploying VM with new plugin, and then telling others that they can download new version of VM, so they could try your new stuff. Now i doubt if you do that, it will take ten minutes or so.
But now you're complaining about the integration APIs (with which I agree). But there is stuff we can do to make this easier, *much* easier. If you've ever looked at how Python deals with this stuff you'll get an idea about how easy that can be - no slang, no VMMaker, no build files other than what is already in Visual Studio. In effect, we should be providing a squeak.h and a squeak.lib and the primitives should look like this:
#include <windows.h> #include <wincred.h> #include "squeak.h"
/* Read a credential from the Windows credential store */ OOP primitiveCredRead(VM *vm, OOP rcvr, OOP args) { Credentials *cred; char *target;
vm->parseArgs(1, "%s", &target); if(!CredRead(targetName, CRED_TYPE_GENERIC, 0,&cred)) return vm->fail(); return vm->stringWithLength(cred->CredentialBlobSize+1, cred->CredentialBlob); }
Voila, done.
<sarcasm> Why Croquet using FFI for OpenGL binding, why not writing a plugin? It would be much more 'stable' and 'robust' and secure. </sarcasm>
Only one reason: The poor integration APIs. I couldn't figure out how to generate proper plugin glue (and later I had lost the source for generating this stufF). And we *paid* for it. Weeks and weeks of obscure crash reports that we couldn't figure out until *finally* we realized that when using client state operations a GC may occur if the rendering is interrupted "just so". We fixed this by (guess what) moving these operations into a plugin.
No objections here. Shit happens. And its really don't matters where: either in language/FFI or in C. You still have to fix that.
It's not so much that shit happens but rather that your sarcastic comment is *completely* wrong and (I think) goes to show how little exposure to the resulting problems you (and pretty much everybody else arguing for that kind of stuff) really have.
Cheers, - Andreas
On 30 September 2010 11:24, Andreas Raab andreas.raab@gmx.de wrote:
On 9/29/2010 6:48 PM, Igor Stasenko wrote:
On 30 September 2010 04:30, Andreas Raabandreas.raab@gmx.de wrote:
See, in my experience that claim is simply wrong. How can you say that (for example) a little test program like the one that I just wrote to experiment with a Windows function would be faster to write using the FFI?
------------------------------ CredTest.cpp ----------------------------
// CredTest.cpp : Defines the entry point for the console application. //
#include "stdafx.h" #include<Windows.h> #include<wincred.h>
LPTSTR readCredential(LPTSTR targetName) { CREDENTIAL *cred;
if(CredRead(targetName, CRED_TYPE_GENERIC, 0,&cred)) { LPTSTR result = (LPTSTR) calloc(cred->CredentialBlobSize+1, sizeof(TCHAR)); memcpy(result, cred->CredentialBlob, cred->CredentialBlobSize); return result; } printf("CredRead failed: %d\n", GetLastError()); return NULL; }
int _tmain(int argc, _TCHAR* argv[]) { LPTSTR credential = readCredential(L"TERMSRV/devmsapps01"); printf("Credential: %s\n", credential); return 0; }
You can imagine that it took me some ten minutes or so (incl. firing up Visual Studio, creating a new project etc) to write this. Now do that in Squeak and keep in mind how much of your time you're wasting in the overhead of declaring the types, functions, defines. That's why I'm saying that unless you can get the equivalent of #include<windows.h> you're not even in the same ballpark.
Wait, in the above you forgot to show me the glue code which exposing this function via some primitive. And you forgot to write the list of steps (in addition to opening Visual studio), how you rebuilding VM using VMMaker , compiling and linking, and finally deploying VM with new plugin, and then telling others that they can download new version of VM, so they could try your new stuff. Now i doubt if you do that, it will take ten minutes or so.
But now you're complaining about the integration APIs (with which I agree). But there is stuff we can do to make this easier, *much* easier. If you've ever looked at how Python deals with this stuff you'll get an idea about how easy that can be - no slang, no VMMaker, no build files other than what is already in Visual Studio. In effect, we should be providing a squeak.h and a squeak.lib and the primitives should look like this:
#include <windows.h> #include <wincred.h> #include "squeak.h"
/* Read a credential from the Windows credential store */ OOP primitiveCredRead(VM *vm, OOP rcvr, OOP args) { Credentials *cred; char *target;
vm->parseArgs(1, "%s", &target); if(!CredRead(targetName, CRED_TYPE_GENERIC, 0,&cred)) return vm->fail(); return vm->stringWithLength(cred->CredentialBlobSize+1, cred->CredentialBlob); }
Voila, done.
Yep. Been there , did that :) I had implemented own automatic code generator which generated bindings for my abandoned smalltalk interpreter. I used SWIG C++ compiler by writing own plugin to it. So, what is stopping us from either use SWIG, or write own C/C++ parser/compiler for automatic generation of external library bindings?
<sarcasm> Why Croquet using FFI for OpenGL binding, why not writing a plugin? It would be much more 'stable' and 'robust' and secure. </sarcasm>
Only one reason: The poor integration APIs. I couldn't figure out how to generate proper plugin glue (and later I had lost the source for generating this stufF). And we *paid* for it. Weeks and weeks of obscure crash reports that we couldn't figure out until *finally* we realized that when using client state operations a GC may occur if the rendering is interrupted "just so". We fixed this by (guess what) moving these operations into a plugin.
No objections here. Shit happens. And its really don't matters where: either in language/FFI or in C. You still have to fix that.
It's not so much that shit happens but rather that your sarcastic comment is *completely* wrong and (I think) goes to show how little exposure to the resulting problems you (and pretty much everybody else arguing for that kind of stuff) really have.
Oh, please. I worked more than 2 years on single big C++ project, in Visual Studio. STL, templates, threads, 10 minutes of compilation time, incredibly hard to move ahead, incredibly hard to reproduce errors (since after 'fix' you were always had to restart program and repeat all steps which may possibly triggered bug). I am sick of C, really sick. If it would be so easy to develop in C, then no one (including me) would even look into smalltalk direction.
Cheers, - Andreas
2010/9/30 Andreas Raab andreas.raab@gmx.de:
On 9/29/2010 6:48 PM, Igor Stasenko wrote:
On 30 September 2010 04:30, Andreas Raabandreas.raab@gmx.de wrote:
See, in my experience that claim is simply wrong. How can you say that (for example) a little test program like the one that I just wrote to experiment with a Windows function would be faster to write using the FFI?
------------------------------ CredTest.cpp ----------------------------
// CredTest.cpp : Defines the entry point for the console application. //
#include "stdafx.h" #include<Windows.h> #include<wincred.h>
LPTSTR readCredential(LPTSTR targetName) { CREDENTIAL *cred;
if(CredRead(targetName, CRED_TYPE_GENERIC, 0,&cred)) { LPTSTR result = (LPTSTR) calloc(cred->CredentialBlobSize+1, sizeof(TCHAR)); memcpy(result, cred->CredentialBlob, cred->CredentialBlobSize); return result; } printf("CredRead failed: %d\n", GetLastError()); return NULL; }
int _tmain(int argc, _TCHAR* argv[]) { LPTSTR credential = readCredential(L"TERMSRV/devmsapps01"); printf("Credential: %s\n", credential); return 0; }
You can imagine that it took me some ten minutes or so (incl. firing up Visual Studio, creating a new project etc) to write this. Now do that in Squeak and keep in mind how much of your time you're wasting in the overhead of declaring the types, functions, defines. That's why I'm saying that unless you can get the equivalent of #include<windows.h> you're not even in the same ballpark.
Wait, in the above you forgot to show me the glue code which exposing this function via some primitive. And you forgot to write the list of steps (in addition to opening Visual studio), how you rebuilding VM using VMMaker , compiling and linking, and finally deploying VM with new plugin, and then telling others that they can download new version of VM, so they could try your new stuff. Now i doubt if you do that, it will take ten minutes or so.
But now you're complaining about the integration APIs (with which I agree). But there is stuff we can do to make this easier, *much* easier. If you've ever looked at how Python deals with this stuff you'll get an idea about how easy that can be - no slang, no VMMaker, no build files other than what is already in Visual Studio. In effect, we should be providing a squeak.h and a squeak.lib and the primitives should look like this:
#include <windows.h> #include <wincred.h> #include "squeak.h"
/* Read a credential from the Windows credential store */ OOP primitiveCredRead(VM *vm, OOP rcvr, OOP args) { Credentials *cred; char *target;
vm->parseArgs(1, "%s", &target); if(!CredRead(targetName, CRED_TYPE_GENERIC, 0,&cred)) return vm->fail(); return vm->stringWithLength(cred->CredentialBlobSize+1, cred->CredentialBlob); }
Voila, done.
<sarcasm> Why Croquet using FFI for OpenGL binding, why not writing a plugin? It would be much more 'stable' and 'robust' and secure. </sarcasm>
Only one reason: The poor integration APIs. I couldn't figure out how to generate proper plugin glue (and later I had lost the source for generating this stufF). And we *paid* for it. Weeks and weeks of obscure crash reports that we couldn't figure out until *finally* we realized that when using client state operations a GC may occur if the rendering is interrupted "just so". We fixed this by (guess what) moving these operations into a plugin.
No objections here. Shit happens. And its really don't matters where: either in language/FFI or in C. You still have to fix that.
It's not so much that shit happens but rather that your sarcastic comment is *completely* wrong and (I think) goes to show how little exposure to the resulting problems you (and pretty much everybody else arguing for that kind of stuff) really have.
Cheers, - Andreas
Agree with some of your arguments, but certainly not the last one: you can't presume of everybody else experience.
From my personnal experience, in the restricted case of simple types
and no #include hell, my productivity did boost in VW with DLLCC compared to writing user primitives. Simply because I didn't have to deal with any C code at all... This may vary with applications for sure.
Nicolas
On 9/30/2010 1:43 AM, Igor Stasenko wrote:
But now you're complaining about the integration APIs (with which I agree). But there is stuff we can do to make this easier, *much* easier. If you've ever looked at how Python deals with this stuff you'll get an idea about how easy that can be - no slang, no VMMaker, no build files other than what is already in Visual Studio. In effect, we should be providing a squeak.h and a squeak.lib and the primitives should look like this:
#include<windows.h> #include<wincred.h> #include "squeak.h"
/* Read a credential from the Windows credential store */ OOP primitiveCredRead(VM *vm, OOP rcvr, OOP args) { Credentials *cred; char *target;
vm->parseArgs(1, "%s",&target); if(!CredRead(targetName, CRED_TYPE_GENERIC, 0,&cred)) returnvm->fail(); return vm->stringWithLength(cred->CredentialBlobSize+1, cred->CredentialBlob); }
Voila, done.
Yep. Been there , did that :) I had implemented own automatic code generator which generated bindings for my abandoned smalltalk interpreter. I used SWIG C++ compiler by writing own plugin to it. So, what is stopping us from either use SWIG, or write own C/C++ parser/compiler for automatic generation of external library bindings?
Someone needs to do it :-)
No objections here. Shit happens. And its really don't matters where: either in language/FFI or in C. You still have to fix that.
It's not so much that shit happens but rather that your sarcastic comment is *completely* wrong and (I think) goes to show how little exposure to the resulting problems you (and pretty much everybody else arguing for that kind of stuff) really have.
Oh, please. I worked more than 2 years on single big C++ project, in Visual Studio. STL, templates, threads, 10 minutes of compilation time, incredibly hard to move ahead, incredibly hard to reproduce errors (since after 'fix' you were always had to restart program and repeat all steps which may possibly triggered bug). I am sick of C, really sick. If it would be so easy to develop in C, then no one (including me) would even look into smalltalk direction.
This isn't my claim. My claim is that for *platform integration tasks* C is superior. Not for general software development. But for a plugin that reads and writes the windows credential store, for a plugin that reads and writes the Apple keychain, for a plugin that interfaces with sockets, for a plugin that deals with platform windows, C/C++/C#/ObjC are the better choices.
Cheers, - Andreas
On 30 September 2010 11:57, Andreas Raab andreas.raab@gmx.de wrote:
Voila, done.
Yep. Been there , did that :) I had implemented own automatic code generator which generated bindings for my abandoned smalltalk interpreter. I used SWIG C++ compiler by writing own plugin to it. So, what is stopping us from either use SWIG, or write own C/C++ parser/compiler for automatic generation of external library bindings?
Someone needs to do it :-)
Amen :)
No objections here. Shit happens. And its really don't matters where: either in language/FFI or in C. You still have to fix that.
It's not so much that shit happens but rather that your sarcastic comment is *completely* wrong and (I think) goes to show how little exposure to the resulting problems you (and pretty much everybody else arguing for that kind of stuff) really have.
Oh, please. I worked more than 2 years on single big C++ project, in Visual Studio. STL, templates, threads, 10 minutes of compilation time, incredibly hard to move ahead, incredibly hard to reproduce errors (since after 'fix' you were always had to restart program and repeat all steps which may possibly triggered bug). I am sick of C, really sick. If it would be so easy to develop in C, then no one (including me) would even look into smalltalk direction.
This isn't my claim. My claim is that for *platform integration tasks* C is superior. Not for general software development. But for a plugin that reads and writes the windows credential store, for a plugin that reads and writes the Apple keychain, for a plugin that interfaces with sockets, for a plugin that deals with platform windows, C/C++/C#/ObjC are the better choices.
How i could argue with that? Of course for interfacing with C, C is better than any other language. :) But its too late. My mind already tainted by smalltalk. I want to do it in smalltalk way. I want to evaluate simple doit and access computer's CMOS memory, as demoed by Gerardo Richarte under SqueakNOS on ESUG conference. I don't want to rebuild whole VM from scratch, each time i need to change something. And you are always need to change something, because we're not living in a perfect world and not running our programs under perfect OS, using perfect VM and libraries and so on.
Cheers, - Andreas
On 30 September 2010 12:11, Igor Stasenko siguctua@gmail.com wrote:
On 30 September 2010 11:57, Andreas Raab andreas.raab@gmx.de wrote:
Voila, done.
Yep. Been there , did that :) I had implemented own automatic code generator which generated bindings for my abandoned smalltalk interpreter. I used SWIG C++ compiler by writing own plugin to it. So, what is stopping us from either use SWIG, or write own C/C++ parser/compiler for automatic generation of external library bindings?
Someone needs to do it :-)
Amen :)
No objections here. Shit happens. And its really don't matters where: either in language/FFI or in C. You still have to fix that.
It's not so much that shit happens but rather that your sarcastic comment is *completely* wrong and (I think) goes to show how little exposure to the resulting problems you (and pretty much everybody else arguing for that kind of stuff) really have.
Oh, please. I worked more than 2 years on single big C++ project, in Visual Studio. STL, templates, threads, 10 minutes of compilation time, incredibly hard to move ahead, incredibly hard to reproduce errors (since after 'fix' you were always had to restart program and repeat all steps which may possibly triggered bug). I am sick of C, really sick. If it would be so easy to develop in C, then no one (including me) would even look into smalltalk direction.
This isn't my claim. My claim is that for *platform integration tasks* C is superior. Not for general software development. But for a plugin that reads and writes the windows credential store, for a plugin that reads and writes the Apple keychain, for a plugin that interfaces with sockets, for a plugin that deals with platform windows, C/C++/C#/ObjC are the better choices.
How i could argue with that? Of course for interfacing with C, C is better than any other language. :)
And yeah, but then you still will need to interface C with smalltalk. And here where fun starts (see VMMaker & slang voodo). So, no matter how much additional C code you write, it makes not a bit easier to use it from smalltalk :)
Of course, automatic library bindings could help with it, but they are also having own limitations. For instance, how you suppose to reflect a C++ class with multiple inheritance in smalltalk? :)
But its too late. My mind already tainted by smalltalk. I want to do it in smalltalk way. I want to evaluate simple doit and access computer's CMOS memory, as demoed by Gerardo Richarte under SqueakNOS on ESUG conference. I don't want to rebuild whole VM from scratch, each time i need to change something. And you are always need to change something, because we're not living in a perfect world and not running our programs under perfect OS, using perfect VM and libraries and so on.
Cheers, - Andreas
-- Best regards, Igor Stasenko AKA sig.
On Thu, Sep 30, 2010 at 01:57:21AM -0700, Andreas Raab wrote:
On 9/30/2010 1:43 AM, Igor Stasenko wrote:
But now you're complaining about the integration APIs (with which I agree). But there is stuff we can do to make this easier, *much* easier. If you've ever looked at how Python deals with this stuff you'll get an idea about how easy that can be - no slang, no VMMaker, no build files other than what is already in Visual Studio. In effect, we should be providing a squeak.h and a squeak.lib and the primitives should look like this:
#include<windows.h> #include<wincred.h> #include "squeak.h"
/* Read a credential from the Windows credential store */ OOP primitiveCredRead(VM *vm, OOP rcvr, OOP args) { Credentials *cred; char *target;
vm->parseArgs(1, "%s",&target); if(!CredRead(targetName, CRED_TYPE_GENERIC, 0,&cred)) returnvm->fail(); return vm->stringWithLength(cred->CredentialBlobSize+1, cred->CredentialBlob); }
Voila, done.
Yep. Been there , did that :) I had implemented own automatic code generator which generated bindings for my abandoned smalltalk interpreter. I used SWIG C++ compiler by writing own plugin to it. So, what is stopping us from either use SWIG, or write own C/C++ parser/compiler for automatic generation of external library bindings?
Someone needs to do it :-)
No objections here. Shit happens. And its really don't matters where: either in language/FFI or in C. You still have to fix that.
It's not so much that shit happens but rather that your sarcastic comment is *completely* wrong and (I think) goes to show how little exposure to the resulting problems you (and pretty much everybody else arguing for that kind of stuff) really have.
Oh, please. I worked more than 2 years on single big C++ project, in Visual Studio. STL, templates, threads, 10 minutes of compilation time, incredibly hard to move ahead, incredibly hard to reproduce errors (since after 'fix' you were always had to restart program and repeat all steps which may possibly triggered bug). I am sick of C, really sick. If it would be so easy to develop in C, then no one (including me) would even look into smalltalk direction.
This isn't my claim. My claim is that for *platform integration tasks* C is superior. Not for general software development. But for a plugin that reads and writes the windows credential store, for a plugin that reads and writes the Apple keychain, for a plugin that interfaces with sockets, for a plugin that deals with platform windows, C/C++/C#/ObjC are the better choices.
There's a certain amount of personal preference involved here, which is a good thing IMHO. OSProcessPlugin, which is pure platform integration, is written entirely in Slang, with some embedded #cCode: but no external C sources. Why? Because I thought it was cool that it was possible to do it in Smalltalk. If I had it to do over again, would I do it the same way? Yes. Does that mean that this approach is better? No, it's just my personal preference.
That said, I do think that there can be some unexpected benefits from writing platform glue code in Smalltalk. I wrote MemoryAccess as a Slang replacement for the low level C macros in the VM, and found to my amazement that the performance of the Slang version was identical to that of the cpp macros (on my system, YMMV). So I ended up thinking that the macros are a performance enhancement in C that actually do not improve performance, and that make the VM code harder to understand. Of course, at the time the macros were done, they were very much needed because the Slang inliner did not originally deliver this kind of performance.
I like the SWIG idea a lot by the way.
$0.02
Dave
+1
----- Original Message ---- From: Andreas Raab andreas.raab@gmx.de To: vm-dev@lists.squeakfoundation.org Sent: Wed, September 29, 2010 6:30:29 PM Subject: Re: [Vm-dev] Cog: A question about: setInterruptCheckChain()
[Side note: Please don't quote entire posts if you're going to reply 2/3rds of the way down. I had to scroll three times through this post to even find what you replied to]
+1
Hi Andreas,
excuse me for flogging the odd limb of a dead horse... I agree with many of your points below but a few I still need to flail away at...
On Wed, Sep 29, 2010 at 4:22 PM, Andreas Raab andreas.raab@gmx.de wrote:
On 9/29/2010 1:03 PM, Eliot Miranda wrote:
I think it /is/, and you shouldn't allow your disinclination for the FFI approach blind you to the fact that one way of keeping the VM small is to produce a really strong flexible FFI that supports threading and callbacks and to implement interfaces above the line. Do you seriously believe things like the ODBC connect should be implemented in plugins?
Absolutely. Memory safety being the prime reason.
I don't see that C provides any more or less memory safety than the FFI, unless the FFI can't access accurate type information for signatures and parameter types, and can't easily manage unmoving objects. I know this is an issue, requiring good header parsing a C compiler modelling tools, or use of things like gcc-xml, but it is possible, and there are clunky solutions to unmoving objects (ExternalData, Alien) and more elegant facilities in gestation (a pinning garbage collector). So if grant me that the FFI can access accurate type information and provide unmoving objects then there's no difference since C allows one to mismanage memory (e.g. free memory still in use, cast a pointer inappropriately, etc, etc) just as much, if not more than, the FFI. Memory safety errors in both C and Smalltalk domains can cause hard-to-dagnose crashes that happen long after the fact. The one advantage that C has is that one has to use only one domain while debugging, C, whereas if one is using the FFI it's likely one has to debug in both.
In the case of event callbacks its a different issue. The current event
queue abstraction is broken in that it prevents Smalltalk code from answering questions asked through events such as WM_QUERYENDSESSION because the VM is forced to queue events, and hence can only provide a default answer in the window proc. So there are areas where having proper callbacks is required to be able to integrate properly with the host operating system.
Please. The issue here is that our event mechanism is a bit simplistic but the Android VM is a great example of how to deal with these issues properly. Sure, callbacks could address this too, but claiming *only* callbacks can address this is simply wrong.
I don't see this in my specific example. One can provide a hack such as specifying a default response before the fact for the VM to respond with to WM_QUERYENDSESSION but that doesn't give the image a chance to act, e.g. to flush data or to query the user. So humour me, how would you solve this specific case without callbacks? The requirement is that the image gets a chance to query the user in response to WM_QUERYENDSESSION so that they can abort the reboot if desired.
I don't disagree with what you say about plugins, or about maintaining
cross-platform abstractions in the VM for core functionality that is common across platforms, but I think its clear one needs true callbacks, and that at least for integrating with the GUI this is the best way to implement it.
A solution that inherently implies exposure of raw pointers to users isn't what I would call the "best" solution. That said, I'm not arguing against callbacks - when you need them you need them. And there are practical situations where it's incredibly handy to have them. And yet, would I design the system so that it depends at its core on such facilities? Hell, no, because in addition to being memory-unsafe, porting an FFI is *much* harder than porting a bit of C support code.
But porting the FFI is /not/ harder than porting a JIT (believe me), and as an FFI grows in sophistication it gets easier to port because techniques and solutions used on one platform are applicable to others (again, I've been there and done that either with discarding the bulk of the assembler in an existing implementation or a new port in the VisualWorks FFI implementations on Alpha AXP, MIPS, PowerPC, SPARC (32-bit & 64-bit) and x86-64).
It really is a matter of where you want to deal with certain aspects
of the system. I prefer to deal with that level of stuff in a plugin via C because:
- it provides a way to do ensure memory safety
there's nothing in plugins per-se to maintain memory safety. A mistake in a plugin is as destructive as a mistake in an FFI call. In either case a wrapper around a particular call can validate and provide safety.
In theory, yes. In practice such wrappers *do* exist in plugins and they do *not* exist in FFI calls. Why? Good question. Probably because people understand that in C land there are more constraints and they are mentally "set up" to keep an eye on the constraints and to make sure the incoming stuff is okay.
I don't agree that they don't exist in Smalltalk code. Well-written Smalltalk code that uses an FFI typically does provide an abstraction-boundary and manage memory behind it. The ODBC connect is a good example.
+ it provides a way to provide abstractions
- so does Smalltalk, and t least in SMalltalk we have inheritance to
enable us to express cross-platform APIs in abstract classes and map these down onto particular platforms in concrete subclasses. This is /far better/ than the sprawling mess that is the platforms tree in the VM.
In theory yes. In practice, who knows. Having looked at some Smalltalks that expose all the platform details (like Dolphin) it seems to me that the equation of "platform FFI code * number of platforms" will be /far more/ of a sprawling mess. Which is probably why (unless I'm mistaken in which case I'll appreciate a correction) precisely none of the cross-platform Smalltalks do that.
Two great advantages that Smalltalk has here are browsability and inheritance. It is straight-forward to use inheritance to group platform fan-out and the IDE supports comparing the various platform-specific implementations. Further, inheritance encourages code-sharing and keeping platform fan-out to a minimum. In contrast the platform tree in the Squeak VM has very little shared code and IME is quite badly organized.
+ the code is faster to write and debug in C
this is debatable.
Hardly. Let's start with "#include <windows.h>" (or whatever platform stuff you happen to need). You're losing right there. The problem is that the system isn't set up to seamlessly interact with C. Even if you've got some auto-magic header file importer that creates structs and defines and stuff, as long as it takes more than, i.e.,
winMessageBox: contents title: title <include: 'windows.h'> MessageBox(NULL, contents, title, MB_OK + MB_ICONWARNING).
you're going to lose because it takes so much longer to get to the point of actually writing the code you're trying to write.
That's a strong point. It was hard in VisualWorks to find the effort to keep the C preprocessor up-to-date. However, I can at least imagine a well-maintained interface to windows.h and know how I would implement it (I've discussed this a lot before; I won't bore you).
Secondly, you *really* need to try the latest Visual Studio and play with its debugger - I was blown away while debugging SqueakSSL when I hit a typical NULL pointer, fixed the code in my plugin DLL, continued in VS and the underlying Squeak KEPT RUNNING! Holy crap. This stuff has come a looooong way.
True. But it breaks my heart. I don't want to be embraced and then extended ;) They're playing catch-up and have probably caught up, but I would still like to compete.
In the VW FFI with the ability to catch exceptions
and pass them back up (doesn't always work, a bad emory corruption may crash the entire system) makes thing easier. But low-level debugging is painful in general.
- access to facilities (threads, interrupts, atomicity) not
available otherwise
Again, a strong FFI provides at least some of these facilities. Not everything can be done through the FFI, but a lot can be done, and elegantly and extensibly, by Smalltalk programmers, not a few VM specialists.
I call BS. The reality is that one needs to know three kinds of things, only one of which would be helped by using the FFI:
- The "mechanics" of writing a plugin, i.e., slang and compilation. This
is the part where the FFI would help, but so would better APIs for marshaling. While it's "cool" that one can run slang plugins in Squeak it is also completely and utterly *useless* when it comes to integrating platform specific stuff and having to deal with Slang, VMMaker and the awful build setups, and then -on top of that- the actual work you're trying to do is just a bit too much.
- Understanding the rules of interfacing the external world, i.e., when
can pointers be kept, what does GC do to your objects etc. None of this is helped by using the FFI.
- Understanding the actual domain code. Again, none of this is helped by
the FFI (to the contrary because nobody will find examples for the weird Smalltalk selectors we use on the web and people will in turn not know where to find the actual value for MB_OK or somesuch).
Basically, the assumption that *all* the difficulty is in writing the marshaling code is nonsense. There is some of it, true, but that is entirely our fault for providing poor integration APIs.
Agreed.
On the downside, there is:
- it's harder to modify for people who don't know C and are not set
up to build a plugin
- it's often aimed at the smallest common denominator (though not
necessarily)
- it is a far more fixed interface that is far harder to evolve
Yes, but when you call it "stable" instead of "fixed" and "robust" instead of "harder to evolve" it becomes a plus :-)
There's plenty of examples of stable interfaces above the FFI. VisualWorks has depended on its database connects; they are its bread if not its butter and are implemented on the FFI. The Windows registry is accessed solely through the FFI, etc. These interfaces are certainly stable and have evolved.
best Eliot
Cheers,
Andreas
Cheers,
- Andreas On the other hand, a VM providing a larger de facto immutable API: - somehow is more secure (a guaranty we can run very old images in very new OSes). - can maintain the illusion that despite the efforts of OS and hardware designers to make it ever more complex, a single personcan still understand (almost) the whole system. - avoid the necessity to embed knowledge of dozens of different variants of different OSes in your image.
Of course, the complexity still exists under the carpet... When I contemplate all the unecessary complex knowledge from those beautiful configure/cmake scripts and macros, I'm not convinced all thiscruft will be easier to modify in Smalltalk than it is in C world.
I perfectly understand too the desire of a VM maintainer to lighten the burden ;) though. Nicolas 2010/9/29 Eliot Miranda<eliot.miranda@gmail.com <mailto:eliot.miranda@gmail.com>>: On Wed, Sep 29, 2010 at 10:22 AM, Andreas Raab<andreas.raab@gmx.de <mailto:andreas.raab@gmx.de>> wrote: On 9/29/2010 9:36 AM, Eliot Miranda wrote: See the recordMouseEvent() and its friends (sqNextEventPut, eventBuffer[1024] ...) in sqWin32Window, for better understanding my point :) Which makes my point well. These are simply maintaining a queue. If one has callbacks then the bulk of the Windows event handling system can be lifted up into the image. This is what Vassili Bykov did in Newspeak above my Alien callbacks. In Newspeak the Windows MainWndProc is a callback: Thus losing any level of platform abstraction. A bad trade-off, IMO. The platform abstraction merely moves from the VM to classes in the image. But it does mean that one can properly implement Windows event semantics (things like query quit events) that the queue prevents. And it means that the VM gets simpler and complexity is handled by better facilities (in Smalltalk with its many advantages for creating and maintaining abstractions with concrete variantimplementations). But you and I have had this discussion a number of times and I take your point about the VM maintaining a black-box abstraction boundary whereas the in-image implementation is in theory porous. As I've said before this doesn't strike me as that important when Smalltalk has had boundary violators such as instVarAt:[put:] for ever and they are rarely abused. If one were to implement a native GUI event interface in Smalltalk using callbacks one would have to police the abstraction boundary. But that's easily done, and there are other benefits - the VM, being simpler, gets more longevity since changing the event interface doesn't imply a VM change - the system, having native interfaces, can support each platform's facilities in full instead of providing the lowest common denominator So IMO it is a good trade-off. best Eliot
Cheers, - Andreas
vm-dev@lists.squeakfoundation.org
-
Andreas Raab -
Bert Freudenberg -
David T. Lewis -
Derek O'Connell -
Eliot Miranda -
Igor Stasenko -
Nicolas Cellier -
Squeak List