On Wed, Sep 29, 2010 at 12:20 AM, Igor Stasenko siguctua@gmail.com wrote:

On 29 September 2010 09:35, Eliot Miranda eliot.miranda@gmail.com wrote:

...

Hi Igor,

On Tue, Sep 28, 2010 at 10:48 PM, Igor Stasenko siguctua@gmail.com

wrote:

...

...

As far as i understood , this is a way to insert own function to force VM to interrupt.

Alas I think it's the reverse of this. It is a way of getting the VM's

heartbeat to interrupt other functions. In both Cog VMs there is a heartbeat (preferrably in a high-priroity thread) that interrupts the VM at approximately 1KHz. It does this by setting the VM's stackLimit to a value that will cause the next stack check to always fail (stacks grow down so (unsigned)-1 is the value). Then the VM checks for events as part of its stack overflow processing. This makes the stack check do double-duty, and avoids having to maintain a counter which (a) is slow due to a read-modify-write cycle and (b) runs at a variable rate because the rate at which the counter counts depends on what the VM is doing. However, the heartbeat depends on the VM performing frequent stack checks (which it does on non-primiive sends) which won't help if for example the VM is executing a long-running primitive.

...

The Bochs plugin executes x86 code generated by the Cog JIT when

simulating the Cog VM. Bochs would happily run forever if the Cog JIT generated code that looped and I'd be hosed, except that setInterruptCheckChain allows the Bochs plugin to stop running when it is interrupted. So this isn't even close to what you want, right?

...

So as i understood by reverse is, that this is the way to force some other (non-main thread) to interrupt itself once heartbeat occurs. Right? Okay, this is userful as well as reverse control: Is there a thread-safe function to force VM to interrupt?

AFAIK its signalExternalSemaphore(), but what if you don't want to signal anything?

For example: i want my plugin to periodically update some small number of objects (probably smallints) held in some array.

...

...

This is cool.

But i don't see another insertion point, which could make this feature

complete.

...

...

Suppose my plugin needs to handle some external events. Then, in my custom interrupt checker i can check if there any pending events to handle, while during interrupt i should be able to install own routine as well, which could actually handle events (like convert events to some object(s) and/or signal semaphore

etc)

...

...

without worrying that i'm doing this in the middle of interpreter cycle, which may damage interpreter's volatile state.

Well, the signalExternalSemaphore mechanism (at least in Cog) is

thread-safe and can be used at any time form any thread to request a signal that will get performed at the next event check.

No, its not the same. During asynchronous event, of course you are limited to semaphores. But during synchronous interrupt you can do much more.

For synchronous activities one could indeed have a simple on-event-check chain. But you could do this with a process waiting on a Delay, polling occasionally, instead of calling some interrupt check function at high frequency every time the VM checks for events. It does that anyway in ioProcessEvents().

What's the use case?

For instance, create a linked list and append new objects directly to

that list once new events available. Then language side won't even need to bother using extra primitive to pool/read events from some source , and can simply do own stuff once new object(s) 'magically' appear in such list.

While one could provide a thread-safe object manipulation mechanism (I wrote a slow prototype in the VisualWorks VM in 1998) I think the best way to do this is with a callback. So Smalltalk code ends up translating from the C data into Smalltalk objects and manipulating the linked list and doing whatever else is needed. Doing otherwise really complicates the garbage collector etc. If instead one focusses on making callbacks as easy and efficient to use and possible one has a general mechanism and overall the system is simpler.

See the recordMouseEvent() and its friends (sqNextEventPut, eventBuffer[1024] ...) in sqWin32Window, for better understanding my point :)

Which makes my point well. These are simply maintaining a queue. If one has callbacks then the bulk of the Windows event handling system can be lifted up into the image. This is what Vassili Bykov did in Newspeak above my Alien callbacks. In Newspeak the Windows MainWndProc is a callback:

protected setupWindowProc = ( windowProc:: api Callback block: [:args :result | result returnInteger: (dispatchMessage: args)] stdcallArgsClass: api WindowProc )

...

The threaded FFI I'm working on will allow one to attempt a callback at

any time. The callback will simply block until it is safe for it to take over the VM. So this will also be thread-safe. But its not one yet.

...

HTH Eliot

P.S. I saw your other messages and will try and respond tomorrow.

Good.

...

...

-- Best regards, Igor Stasenko AKA sig.

-- Best regards, Igor Stasenko AKA sig.

On Wed, Sep 29, 2010 at 10:22 AM, Andreas Raab andreas.raab@gmx.de wrote:

On 9/29/2010 9:36 AM, Eliot Miranda wrote:

...

See the recordMouseEvent() and its friends (sqNextEventPut, eventBuffer[1024] ...) in sqWin32Window, for better understanding my point :)

Which makes my point well. These are simply maintaining a queue. If one has callbacks then the bulk of the Windows event handling system can be lifted up into the image. This is what Vassili Bykov did in Newspeak above my Alien callbacks. In Newspeak the Windows MainWndProc is a callback:

Thus losing any level of platform abstraction. A bad trade-off, IMO.

The platform abstraction merely moves from the VM to classes in the image. But it does mean that one can properly implement Windows event semantics (things like query quit events) that the queue prevents. And it means that the VM gets simpler and complexity is handled by better facilities (in Smalltalk with its many advantages for creating and maintaining abstractions with concrete variant implementations).

But you and I have had this discussion a number of times and I take your point about the VM maintaining a black-box abstraction boundary whereas the in-image implementation is in theory porous. As I've said before this doesn't strike me as that important when Smalltalk has had boundary violators such as instVarAt:[put:] for ever and they are rarely abused. If one were to implement a native GUI event interface in Smalltalk using callbacks one would have to police the abstraction boundary. But that's easily done, and there are other benefits - the VM, being simpler, gets more longevity since changing the event interface doesn't imply a VM change - the system, having native interfaces, can support each platform's facilities in full instead of providing the lowest common denominator

So IMO it is a good trade-off.

best Eliot

Cheers,

• Andreas

On 9/29/2010 11:53 AM, Nicolas Cellier wrote:

Another aspect of the trade-off is releasing creativity. To change the VM you need advanced skills in both:

• Smalltalk VM internals
• and external libraries you want to use.

That resticts dramatically the list of creators.

Plus, the difficulties for sharing your creation, you shall distribute:

• either a binary version, but this means users cannot have a mixture

of different features if those require a core modification,

• or VMMaker patches but your audience will shrink again.

That's why we have plugins. Keeping the core of the VM as small as possible is most definitely a Very Good Thing and not what's being discussed here. What's being discussed here is if (from that point on) one should provide layers of abstractions via plugins or if one should just expose the image directly to all of the idiosyncrasies of the platform, handing it pointers and let it make FFI calls.

It really is a matter of where you want to deal with certain aspects of the system. I prefer to deal with that level of stuff in a plugin via C because: + it provides a way to do ensure memory safety + it provides a way to provide abstractions + the code is faster to write and debug in C + access to facilities (threads, interrupts, atomicity) not available otherwise

On the downside, there is: - it's harder to modify for people who don't know C and are not set up to build a plugin - it's often aimed at the smallest common denominator (though not necessarily)

Cheers, - Andreas

On the other hand, a VM providing a larger de facto immutable API:

• somehow is more secure (a guaranty we can run very old images in

very new OSes).

• can maintain the illusion that despite the efforts of OS and

hardware designers to make it ever more complex, a single person can still understand (almost) the whole system.

• avoid the necessity to embed knowledge of dozens of different

variants of different OSes in your image.

Of course, the complexity still exists under the carpet... When I contemplate all the unecessary complex knowledge from those beautiful configure/cmake scripts and macros, I'm not convinced all this cruft will be easier to modify in Smalltalk than it is in C world.

I perfectly understand too the desire of a VM maintainer to lighten the burden ;) though.

Nicolas

2010/9/29 Eliot Mirandaeliot.miranda@gmail.com:

...

On Wed, Sep 29, 2010 at 10:22 AM, Andreas Raabandreas.raab@gmx.de wrote:

...

On 9/29/2010 9:36 AM, Eliot Miranda wrote:

...

See the recordMouseEvent() and its friends (sqNextEventPut,
eventBuffer[1024] ...) in sqWin32Window, for better understanding my
point :)

Which makes my point well. These are simply maintaining a queue. If one has callbacks then the bulk of the Windows event handling system can be lifted up into the image. This is what Vassili Bykov did in Newspeak above my Alien callbacks. In Newspeak the Windows MainWndProc is a callback:

Thus losing any level of platform abstraction. A bad trade-off, IMO.

The platform abstraction merely moves from the VM to classes in the image. But it does mean that one can properly implement Windows event semantics (things like query quit events) that the queue prevents. And it means that the VM gets simpler and complexity is handled by better facilities (in Smalltalk with its many advantages for creating and maintaining abstractions with concrete variant implementations). But you and I have had this discussion a number of times and I take your point about the VM maintaining a black-box abstraction boundary whereas the in-image implementation is in theory porous. As I've said before this doesn't strike me as that important when Smalltalk has had boundary violators such as instVarAt:[put:] for ever and they are rarely abused. If one were to implement a native GUI event interface in Smalltalk using callbacks one would have to police the abstraction boundary. But that's easily done, and there are other benefits

• the VM, being simpler, gets more longevity since changing the event interface doesn't imply a VM change
• the system, having native interfaces, can support each platform's facilities in full instead of providing the lowest common denominator

So IMO it is a good trade-off. best Eliot

...

Cheers,

• Andreas

On 29 September 2010 23:03, Eliot Miranda eliot.miranda@gmail.com wrote:

On Wed, Sep 29, 2010 at 12:22 PM, Andreas Raab andreas.raab@gmx.de wrote:

...

On 9/29/2010 11:53 AM, Nicolas Cellier wrote:

...

Another aspect of the trade-off is releasing creativity. To change the VM you need advanced skills in both:

• Smalltalk VM internals
• and external libraries you want to use.

That resticts dramatically the list of creators.

Plus, the difficulties for sharing your creation, you shall distribute:

• either a binary version, but this means users cannot have a mixture

of different features if those require a core modification,

• or VMMaker patches but your audience will shrink again.

That's why we have plugins. Keeping the core of the VM as small as possible is most definitely a Very Good Thing and not what's being discussed here.

I think it /is/, and you shouldn't allow your disinclination for the FFI approach blind you to the fact that one way of keeping the VM small is to produce a really strong flexible FFI that supports threading and callbacks and to implement interfaces above the line.  Do you seriously believe things like the ODBC connect should be implemented in plugins?

...

What's being discussed here is if (from that point on) one should provide layers of abstractions via plugins or if one should just expose the image directly to all of the idiosyncrasies of the platform, handing it pointers and let it make FFI calls.

In the case of event callbacks its a different issue.  The current event queue abstraction is broken in that it prevents Smalltalk code from answering questions asked through events such as WM_QUERYENDSESSION because the VM is forced to queue events, and hence can only provide a default answer in the window proc.  So there are areas where having proper callbacks is required to be able to integrate properly with the host operating system. I don't disagree with what you say about plugins, or about maintaining cross-platform abstractions in the VM for core functionality that is common across platforms, but I think its clear one needs true callbacks, and that at least for integrating with the GUI this is the best way to implement it.

...

It really is a matter of where you want to deal with certain aspects of the system. I prefer to deal with that level of stuff in a plugin via C because:

• it provides a way to do ensure memory safety

there's nothing in plugins per-se to maintain memory safety.  A mistake in a plugin is as destructive as a mistake in an FFI call.  In either case a wrapper around a particular call can validate and provide safety.

...

• it provides a way to provide abstractions

• so does Smalltalk, and t least in SMalltalk we have inheritance to enable us to express cross-platform APIs in abstract classes and map these down onto particular platforms in concrete subclasses.  This is /far better/ than the sprawling mess that is the platforms tree in the VM.

...

• the code is faster to write and debug in C

this is debatable.  In the VW FFI with the ability to catch exceptions and pass them back up (doesn't always work, a bad emory corruption may crash the entire system) makes thing easier.  But low-level debugging is painful in general.

...

• access to facilities (threads, interrupts, atomicity) not available otherwise

Again, a strong FFI provides at least some of these facilities.  Not everything can be done through the FFI, but a lot can be done, and elegantly and extensibly, by Smalltalk programmers, not a few VM specialists.

...

On the downside, there is:

• it's harder to modify for people who don't know C and are not set up to build a plugin
• it's often aimed at the smallest common denominator (though not necessarily)

• it is a far more fixed interface that is far harder to evolve

Eliot, i don't know what to say in addition. I have same thoughts as you.

Its not a question, that VM should provide a default platform abstraction layer(s). But the question, how i can make part's of it optional, or easily alter/replace them with own implementation (either plugin of ffi) in case of need, without the need of shipping of own, custom built VM?

Where ends a VM's mandatory functionality and starts an optional one? a) providing UI & event handling? Apparently not for those who wants to run VM as GUI-less command-line tool or implement own GUI framework. b) providing a network layer? Apparently not for those, who not using network in their application(s), and instead want to make sure that no network activity possible. Or again, wants to provide own network layer.

FFI is not a silver bullet. But my point is not putting you before choice: use C or FFI. Its about making VM more flexible at run-time, not at compile time.

best, Eliot

On 9/29/2010 1:03 PM, Eliot Miranda wrote:

I think it /is/, and you shouldn't allow your disinclination for the FFI approach blind you to the fact that one way of keeping the VM small is to produce a really strong flexible FFI that supports threading and callbacks and to implement interfaces above the line. Do you seriously believe things like the ODBC connect should be implemented in plugins?

Absolutely. Memory safety being the prime reason.

In the case of event callbacks its a different issue. The current event queue abstraction is broken in that it prevents Smalltalk code from answering questions asked through events such as WM_QUERYENDSESSION because the VM is forced to queue events, and hence can only provide a default answer in the window proc. So there are areas where having proper callbacks is required to be able to integrate properly with the host operating system.

Please. The issue here is that our event mechanism is a bit simplistic but the Android VM is a great example of how to deal with these issues properly. Sure, callbacks could address this too, but claiming *only* callbacks can address this is simply wrong.

I don't disagree with what you say about plugins, or about maintaining cross-platform abstractions in the VM for core functionality that is common across platforms, but I think its clear one needs true callbacks, and that at least for integrating with the GUI this is the best way to implement it.

A solution that inherently implies exposure of raw pointers to users isn't what I would call the "best" solution. That said, I'm not arguing against callbacks - when you need them you need them. And there are practical situations where it's incredibly handy to have them. And yet, would I design the system so that it depends at its core on such facilities? Hell, no, because in addition to being memory-unsafe, porting an FFI is *much* harder than porting a bit of C support code.

It really is a matter of where you want to deal with certain aspects
of the system. I prefer to deal with that level of stuff in a plugin
via C because:
+ it provides a way to do ensure memory safety

there's nothing in plugins per-se to maintain memory safety. A mistake in a plugin is as destructive as a mistake in an FFI call. In either case a wrapper around a particular call can validate and provide safety.

In theory, yes. In practice such wrappers *do* exist in plugins and they do *not* exist in FFI calls. Why? Good question. Probably because people understand that in C land there are more constraints and they are mentally "set up" to keep an eye on the constraints and to make sure the incoming stuff is okay.

+ it provides a way to provide abstractions

• so does Smalltalk, and t least in SMalltalk we have inheritance to

enable us to express cross-platform APIs in abstract classes and map these down onto particular platforms in concrete subclasses. This is /far better/ than the sprawling mess that is the platforms tree in the VM.

In theory yes. In practice, who knows. Having looked at some Smalltalks that expose all the platform details (like Dolphin) it seems to me that the equation of "platform FFI code * number of platforms" will be /far more/ of a sprawling mess. Which is probably why (unless I'm mistaken in which case I'll appreciate a correction) precisely none of the cross-platform Smalltalks do that.

+ the code is faster to write and debug in C

this is debatable.

Hardly. Let's start with "#include <windows.h>" (or whatever platform stuff you happen to need). You're losing right there. The problem is that the system isn't set up to seamlessly interact with C. Even if you've got some auto-magic header file importer that creates structs and defines and stuff, as long as it takes more than, i.e.,

winMessageBox: contents title: title <include: 'windows.h'> MessageBox(NULL, contents, title, MB_OK + MB_ICONWARNING).

you're going to lose because it takes so much longer to get to the point of actually writing the code you're trying to write.

Secondly, you *really* need to try the latest Visual Studio and play with its debugger - I was blown away while debugging SqueakSSL when I hit a typical NULL pointer, fixed the code in my plugin DLL, continued in VS and the underlying Squeak KEPT RUNNING! Holy crap. This stuff has come a looooong way.

In the VW FFI with the ability to catch exceptions and pass them back up (doesn't always work, a bad emory corruption may crash the entire system) makes thing easier. But low-level debugging is painful in general.

+ access to facilities (threads, interrupts, atomicity) not
available otherwise

Again, a strong FFI provides at least some of these facilities. Not everything can be done through the FFI, but a lot can be done, and elegantly and extensibly, by Smalltalk programmers, not a few VM specialists.

I call BS. The reality is that one needs to know three kinds of things, only one of which would be helped by using the FFI:

1) The "mechanics" of writing a plugin, i.e., slang and compilation. This is the part where the FFI would help, but so would better APIs for marshaling. While it's "cool" that one can run slang plugins in Squeak it is also completely and utterly *useless* when it comes to integrating platform specific stuff and having to deal with Slang, VMMaker and the awful build setups, and then -on top of that- the actual work you're trying to do is just a bit too much.

2) Understanding the rules of interfacing the external world, i.e., when can pointers be kept, what does GC do to your objects etc. None of this is helped by using the FFI.

3) Understanding the actual domain code. Again, none of this is helped by the FFI (to the contrary because nobody will find examples for the weird Smalltalk selectors we use on the web and people will in turn not know where to find the actual value for MB_OK or somesuch).

Basically, the assumption that *all* the difficulty is in writing the marshaling code is nonsense. There is some of it, true, but that is entirely our fault for providing poor integration APIs.

On the downside, there is:
- it's harder to modify for people who don't know C and are not set
up to build a plugin
- it's often aimed at the smallest common denominator (though not
necessarily)

• it is a far more fixed interface that is far harder to evolve

Yes, but when you call it "stable" instead of "fixed" and "robust" instead of "harder to evolve" it becomes a plus :-)

Cheers, - Andreas

Cheers,
  - Andreas


    On the other hand, a VM providing a larger de facto immutable API:
    - somehow is more secure (a guaranty we can run very old images in
    very new OSes).
    - can maintain the illusion that despite the efforts of OS and
    hardware designers to make it ever more complex, a single person can
    still understand (almost) the whole system.
    - avoid the necessity to embed knowledge of dozens of different
    variants of different OSes in your image.

    Of course, the complexity still exists under the carpet... When I
    contemplate all the unecessary complex knowledge from those
    beautiful
    configure/cmake scripts and macros, I'm not convinced all this cruft
    will be easier to modify in Smalltalk than it is in C world.

    I perfectly understand too the desire of a VM maintainer to lighten
    the burden ;) though.

    Nicolas

    2010/9/29 Eliot Miranda<eliot.miranda@gmail.com
    <mailto:eliot.miranda@gmail.com>>:




        On Wed, Sep 29, 2010 at 10:22 AM, Andreas
        Raab<andreas.raab@gmx.de <mailto:andreas.raab@gmx.de>>  wrote:


            On 9/29/2010 9:36 AM, Eliot Miranda wrote:


                    See the recordMouseEvent() and its friends
                (sqNextEventPut,
                    eventBuffer[1024] ...) in sqWin32Window, for
                better understanding my
                    point :)


                Which makes my point well.  These are simply
                maintaining a queue.  If
                one has callbacks then the bulk of the Windows event
                handling system can
                be lifted up into the image.  This is what Vassili
                Bykov did in Newspeak
                above my Alien callbacks.  In Newspeak the Windows
                MainWndProc is a
                callback:


            Thus losing any level of platform abstraction. A bad
            trade-off, IMO.


        The platform abstraction merely moves from the VM to classes
        in the image.  But it does mean that one can properly
        implement Windows event semantics (things like query quit
        events) that the queue prevents.  And it means that the VM
        gets simpler and complexity is handled by better facilities
        (in Smalltalk with its many advantages for creating and
        maintaining abstractions with concrete variant implementations).
        But you and I have had this discussion a number of times and
        I take your point about the VM maintaining a black-box
        abstraction boundary whereas the in-image implementation is
        in theory porous.  As I've said before this doesn't strike
        me as that important when Smalltalk has had boundary
        violators such as instVarAt:[put:] for ever and they are
        rarely abused.  If one were to implement a native GUI event
        interface in Smalltalk using callbacks one would have to
        police the abstraction boundary.  But that's easily done,
        and there are other benefits
        - the VM, being simpler, gets more longevity since changing
        the event interface doesn't imply a VM change
        - the system, having native interfaces, can support each
        platform's facilities in full instead of providing the
        lowest common denominator
        So IMO it is a good trade-off.
        best
        Eliot


            Cheers,
              - Andreas

[Side note: Please don't quote entire posts if you're going to reply 2/3rds of the way down. I had to scroll three times through this post to even find what you replied to]

On 9/29/2010 5:50 PM, Igor Stasenko wrote:

...

1. The "mechanics" of writing a plugin, i.e., slang and compilation. This is

the part where the FFI would help, but so would better APIs for marshaling. While it's "cool" that one can run slang plugins in Squeak it is also completely and utterly *useless* when it comes to integrating platform specific stuff and having to deal with Slang, VMMaker and the awful build setups, and then -on top of that- the actual work you're trying to do is just a bit too much.

And i am here looking for a way to make it less painful. If one can make own stuff to work with OS via FFI, why he needs to go into slang , VMMaker, cmake, gcc jungles?

One word: Memory safety. Oh, that's two :-)

Probably, when he consider that FFI is good, but have to use more safer solution, then he free to go and implement plugin or change VM, whatever. But for prototyping and fast implementation, FFI much better.

See, in my experience that claim is simply wrong. How can you say that (for example) a little test program like the one that I just wrote to experiment with a Windows function would be faster to write using the FFI?

------------------------------ CredTest.cpp ----------------------------

// CredTest.cpp : Defines the entry point for the console application. //

#include "stdafx.h" #include <Windows.h> #include <wincred.h>

LPTSTR readCredential(LPTSTR targetName) { CREDENTIAL *cred;

if(CredRead(targetName, CRED_TYPE_GENERIC, 0, &cred)) { LPTSTR result = (LPTSTR) calloc(cred->CredentialBlobSize+1, sizeof(TCHAR)); memcpy(result, cred->CredentialBlob, cred->CredentialBlobSize); return result; } printf("CredRead failed: %d\n", GetLastError()); return NULL; }

int _tmain(int argc, _TCHAR* argv[]) { LPTSTR credential = readCredential(L"TERMSRV/devmsapps01"); printf("Credential: %s\n", credential); return 0; } ----------------------------------------------------------------------

You can imagine that it took me some ten minutes or so (incl. firing up Visual Studio, creating a new project etc) to write this. Now do that in Squeak and keep in mind how much of your time you're wasting in the overhead of declaring the types, functions, defines. That's why I'm saying that unless you can get the equivalent of #include <windows.h> you're not even in the same ballpark.

<sarcasm> Why Croquet using FFI for OpenGL binding, why not writing a plugin? It would be much more 'stable' and 'robust' and secure. </sarcasm>

Only one reason: The poor integration APIs. I couldn't figure out how to generate proper plugin glue (and later I had lost the source for generating this stufF). And we *paid* for it. Weeks and weeks of obscure crash reports that we couldn't figure out until *finally* we realized that when using client state operations a GC may occur if the rendering is interrupted "just so". We fixed this by (guess what) moving these operations into a plugin.

Cheers, - Andreas

On 9/29/2010 6:48 PM, Igor Stasenko wrote:

On 30 September 2010 04:30, Andreas Raabandreas.raab@gmx.de wrote:

...

See, in my experience that claim is simply wrong. How can you say that (for example) a little test program like the one that I just wrote to experiment with a Windows function would be faster to write using the FFI?

------------------------------ CredTest.cpp ----------------------------

// CredTest.cpp : Defines the entry point for the console application. //

#include "stdafx.h" #include<Windows.h> #include<wincred.h>

LPTSTR readCredential(LPTSTR targetName) { CREDENTIAL *cred;

    if(CredRead(targetName, CRED_TYPE_GENERIC, 0,&cred)) {
            LPTSTR result = (LPTSTR) calloc(cred->CredentialBlobSize+1,

sizeof(TCHAR)); memcpy(result, cred->CredentialBlob, cred->CredentialBlobSize); return result; } printf("CredRead failed: %d\n", GetLastError()); return NULL; }

int _tmain(int argc, _TCHAR* argv[]) { LPTSTR credential = readCredential(L"TERMSRV/devmsapps01"); printf("Credential: %s\n", credential); return 0; }

---

You can imagine that it took me some ten minutes or so (incl. firing up Visual Studio, creating a new project etc) to write this. Now do that in Squeak and keep in mind how much of your time you're wasting in the overhead of declaring the types, functions, defines. That's why I'm saying that unless you can get the equivalent of #include<windows.h> you're not even in the same ballpark.

Wait, in the above you forgot to show me the glue code which exposing this function via some primitive. And you forgot to write the list of steps (in addition to opening Visual studio), how you rebuilding VM using VMMaker , compiling and linking, and finally deploying VM with new plugin, and then telling others that they can download new version of VM, so they could try your new stuff. Now i doubt if you do that, it will take ten minutes or so.

But now you're complaining about the integration APIs (with which I agree). But there is stuff we can do to make this easier, *much* easier. If you've ever looked at how Python deals with this stuff you'll get an idea about how easy that can be - no slang, no VMMaker, no build files other than what is already in Visual Studio. In effect, we should be providing a squeak.h and a squeak.lib and the primitives should look like this:

#include <windows.h> #include <wincred.h> #include "squeak.h"

/* Read a credential from the Windows credential store */ OOP primitiveCredRead(VM *vm, OOP rcvr, OOP args) { Credentials *cred; char *target;

vm->parseArgs(1, "%s", &target); if(!CredRead(targetName, CRED_TYPE_GENERIC, 0,&cred)) return vm->fail(); return vm->stringWithLength(cred->CredentialBlobSize+1, cred->CredentialBlob); }

Voila, done.

...

...

<sarcasm> Why Croquet using FFI for OpenGL binding, why not writing a plugin? It would be much more 'stable' and 'robust' and secure. </sarcasm>

Only one reason: The poor integration APIs. I couldn't figure out how to generate proper plugin glue (and later I had lost the source for generating this stufF). And we *paid* for it. Weeks and weeks of obscure crash reports that we couldn't figure out until *finally* we realized that when using client state operations a GC may occur if the rendering is interrupted "just so". We fixed this by (guess what) moving these operations into a plugin.

No objections here. Shit happens. And its really don't matters where: either in language/FFI or in C. You still have to fix that.

It's not so much that shit happens but rather that your sarcastic comment is *completely* wrong and (I think) goes to show how little exposure to the resulting problems you (and pretty much everybody else arguing for that kind of stuff) really have.

Cheers, - Andreas

On 9/30/2010 1:43 AM, Igor Stasenko wrote:

...

But now you're complaining about the integration APIs (with which I agree). But there is stuff we can do to make this easier, *much* easier. If you've ever looked at how Python deals with this stuff you'll get an idea about how easy that can be - no slang, no VMMaker, no build files other than what is already in Visual Studio. In effect, we should be providing a squeak.h and a squeak.lib and the primitives should look like this:

#include<windows.h> #include<wincred.h> #include "squeak.h"

/* Read a credential from the Windows credential store */ OOP primitiveCredRead(VM *vm, OOP rcvr, OOP args) { Credentials *cred; char *target;

    vm->parseArgs(1, "%s",&target);
    if(!CredRead(targetName, CRED_TYPE_GENERIC, 0,&cred)) return

vm->fail(); return vm->stringWithLength(cred->CredentialBlobSize+1, cred->CredentialBlob); }

Voila, done.

Yep. Been there , did that :) I had implemented own automatic code generator which generated bindings for my abandoned smalltalk interpreter. I used SWIG C++ compiler by writing own plugin to it. So, what is stopping us from either use SWIG, or write own C/C++ parser/compiler for automatic generation of external library bindings?

Someone needs to do it :-)

...

...

No objections here. Shit happens. And its really don't matters where: either in language/FFI or in C. You still have to fix that.

It's not so much that shit happens but rather that your sarcastic comment is *completely* wrong and (I think) goes to show how little exposure to the resulting problems you (and pretty much everybody else arguing for that kind of stuff) really have.

Oh, please. I worked more than 2 years on single big C++ project, in Visual Studio. STL, templates, threads, 10 minutes of compilation time, incredibly hard to move ahead, incredibly hard to reproduce errors (since after 'fix' you were always had to restart program and repeat all steps which may possibly triggered bug). I am sick of C, really sick. If it would be so easy to develop in C, then no one (including me) would even look into smalltalk direction.

This isn't my claim. My claim is that for *platform integration tasks* C is superior. Not for general software development. But for a plugin that reads and writes the windows credential store, for a plugin that reads and writes the Apple keychain, for a plugin that interfaces with sockets, for a plugin that deals with platform windows, C/C++/C#/ObjC are the better choices.

Cheers, - Andreas

On 30 September 2010 12:11, Igor Stasenko siguctua@gmail.com wrote:

On 30 September 2010 11:57, Andreas Raab andreas.raab@gmx.de wrote:

...

...

...

Voila, done.

Yep. Been there , did that :) I had implemented own automatic code generator which generated bindings for my abandoned smalltalk interpreter. I used SWIG C++ compiler by writing own plugin to it. So, what is stopping us from either use SWIG, or write own C/C++ parser/compiler for automatic generation of external library bindings?

Someone needs to do it :-)

Amen :)

...

...

...

...

No objections here. Shit happens. And its really don't matters where: either in language/FFI or in C. You still have to fix that.

It's not so much that shit happens but rather that your sarcastic comment is *completely* wrong and (I think) goes to show how little exposure to the resulting problems you (and pretty much everybody else arguing for that kind of stuff) really have.

Oh, please. I worked more than 2 years on single big C++ project, in Visual Studio. STL, templates, threads, 10 minutes of compilation time, incredibly hard to move ahead, incredibly hard to reproduce errors (since after 'fix' you were always had to restart program and repeat all steps which may possibly triggered bug). I am sick of C, really sick. If it would be so easy to develop in C, then no one (including me) would even look into smalltalk direction.

This isn't my claim. My claim is that for *platform integration tasks* C is superior. Not for general software development. But for a plugin that reads and writes the windows credential store, for a plugin that reads and writes the Apple keychain, for a plugin that interfaces with sockets, for a plugin that deals with platform windows, C/C++/C#/ObjC are the better choices.

How i could argue with that? Of course for interfacing with C, C is better than any other language. :)

And yeah, but then you still will need to interface C with smalltalk. And here where fun starts (see VMMaker & slang voodo). So, no matter how much additional C code you write, it makes not a bit easier to use it from smalltalk :)

Of course, automatic library bindings could help with it, but they are also having own limitations. For instance, how you suppose to reflect a C++ class with multiple inheritance in smalltalk? :)

But its too late. My mind already tainted by smalltalk. I want to do it in smalltalk way. I want to evaluate simple doit and access computer's CMOS memory, as demoed by Gerardo Richarte under SqueakNOS on ESUG conference. I don't want to rebuild whole VM from scratch, each time i need to change something. And you are always need to change something, because we're not living in a perfect world and not running our programs under perfect OS, using perfect VM and libraries and so on.

...

Cheers,  - Andreas

-- Best regards, Igor Stasenko AKA sig.

2010/9/30 Andreas Raab andreas.raab@gmx.de:

On 9/29/2010 6:48 PM, Igor Stasenko wrote:

...

On 30 September 2010 04:30, Andreas Raabandreas.raab@gmx.de  wrote:

...

See, in my experience that claim is simply wrong. How can you say that (for example) a little test program like the one that I just wrote to experiment with a Windows function would be faster to write using the FFI?

------------------------------ CredTest.cpp ----------------------------

// CredTest.cpp : Defines the entry point for the console application. //

#include "stdafx.h" #include<Windows.h> #include<wincred.h>

LPTSTR readCredential(LPTSTR targetName) {        CREDENTIAL *cred;

if(CredRead(targetName, CRED_TYPE_GENERIC, 0,&cred)) {                LPTSTR result = (LPTSTR) calloc(cred->CredentialBlobSize+1, sizeof(TCHAR));                memcpy(result, cred->CredentialBlob, cred->CredentialBlobSize);                return result;        }        printf("CredRead failed: %d\n", GetLastError());        return NULL; }

int _tmain(int argc, _TCHAR* argv[]) {        LPTSTR credential = readCredential(L"TERMSRV/devmsapps01");        printf("Credential: %s\n", credential);        return 0; }

---

You can imagine that it took me some ten minutes or so (incl. firing up Visual Studio, creating a new project etc) to write this. Now do that in Squeak and keep in mind how much of your time you're wasting in the overhead of declaring the types, functions, defines. That's why I'm saying that unless you can get the equivalent of #include<windows.h>  you're not even in the same ballpark.

Wait, in the above you forgot to show me the glue code which exposing this function via some primitive. And you forgot to write the list of steps (in addition to opening Visual studio), how you rebuilding VM using VMMaker , compiling and linking, and finally deploying VM with new plugin, and then telling others that they can download new version of VM, so they could try your new stuff. Now i doubt if you do that, it will take ten minutes or so.

But now you're complaining about the integration APIs (with which I agree). But there is stuff we can do to make this easier, *much* easier. If you've ever looked at how Python deals with this stuff you'll get an idea about how easy that can be - no slang, no VMMaker, no build files other than what is already in Visual Studio. In effect, we should be providing a squeak.h and a squeak.lib and the primitives should look like this:

#include <windows.h> #include <wincred.h> #include "squeak.h"

/* Read a credential from the Windows credential store */ OOP primitiveCredRead(VM *vm, OOP rcvr, OOP args) {        Credentials *cred;        char *target;

vm->parseArgs(1, "%s", &target);        if(!CredRead(targetName, CRED_TYPE_GENERIC, 0,&cred)) return vm->fail();        return vm->stringWithLength(cred->CredentialBlobSize+1, cred->CredentialBlob); }

Voila, done.

...

...

...

<sarcasm> Why Croquet using FFI for OpenGL binding, why not writing a plugin? It would be much more 'stable' and 'robust' and secure. </sarcasm>

Only one reason: The poor integration APIs. I couldn't figure out how to generate proper plugin glue (and later I had lost the source for generating this stufF). And we *paid* for it. Weeks and weeks of obscure crash reports that we couldn't figure out until *finally* we realized that when using client state operations a GC may occur if the rendering is interrupted "just so". We fixed this by (guess what) moving these operations into a plugin.

No objections here. Shit happens. And its really don't matters where: either in language/FFI or in C. You still have to fix that.

It's not so much that shit happens but rather that your sarcastic comment is *completely* wrong and (I think) goes to show how little exposure to the resulting problems you (and pretty much everybody else arguing for that kind of stuff) really have.

Cheers,  - Andreas

Agree with some of your arguments, but certainly not the last one: you can't presume of everybody else experience.

From my personnal experience, in the restricted case of simple types

and no #include hell, my productivity did boost in VW with DLLCC compared to writing user primitives. Simply because I didn't have to deal with any C code at all... This may vary with applications for sure.

Nicolas

Hi Andreas,

excuse me for flogging the odd limb of a dead horse... I agree with many of your points below but a few I still need to flail away at...

On Wed, Sep 29, 2010 at 4:22 PM, Andreas Raab andreas.raab@gmx.de wrote:

On 9/29/2010 1:03 PM, Eliot Miranda wrote:

...

I think it /is/, and you shouldn't allow your disinclination for the FFI approach blind you to the fact that one way of keeping the VM small is to produce a really strong flexible FFI that supports threading and callbacks and to implement interfaces above the line. Do you seriously believe things like the ODBC connect should be implemented in plugins?

Absolutely. Memory safety being the prime reason.

I don't see that C provides any more or less memory safety than the FFI, unless the FFI can't access accurate type information for signatures and parameter types, and can't easily manage unmoving objects. I know this is an issue, requiring good header parsing a C compiler modelling tools, or use of things like gcc-xml, but it is possible, and there are clunky solutions to unmoving objects (ExternalData, Alien) and more elegant facilities in gestation (a pinning garbage collector). So if grant me that the FFI can access accurate type information and provide unmoving objects then there's no difference since C allows one to mismanage memory (e.g. free memory still in use, cast a pointer inappropriately, etc, etc) just as much, if not more than, the FFI. Memory safety errors in both C and Smalltalk domains can cause hard-to-dagnose crashes that happen long after the fact. The one advantage that C has is that one has to use only one domain while debugging, C, whereas if one is using the FFI it's likely one has to debug in both.

In the case of event callbacks its a different issue. The current event

...

queue abstraction is broken in that it prevents Smalltalk code from answering questions asked through events such as WM_QUERYENDSESSION because the VM is forced to queue events, and hence can only provide a default answer in the window proc. So there are areas where having proper callbacks is required to be able to integrate properly with the host operating system.

Please. The issue here is that our event mechanism is a bit simplistic but the Android VM is a great example of how to deal with these issues properly. Sure, callbacks could address this too, but claiming *only* callbacks can address this is simply wrong.

I don't see this in my specific example. One can provide a hack such as specifying a default response before the fact for the VM to respond with to WM_QUERYENDSESSION but that doesn't give the image a chance to act, e.g. to flush data or to query the user. So humour me, how would you solve this specific case without callbacks? The requirement is that the image gets a chance to query the user in response to WM_QUERYENDSESSION so that they can abort the reboot if desired.

I don't disagree with what you say about plugins, or about maintaining

...

cross-platform abstractions in the VM for core functionality that is common across platforms, but I think its clear one needs true callbacks, and that at least for integrating with the GUI this is the best way to implement it.

A solution that inherently implies exposure of raw pointers to users isn't what I would call the "best" solution. That said, I'm not arguing against callbacks - when you need them you need them. And there are practical situations where it's incredibly handy to have them. And yet, would I design the system so that it depends at its core on such facilities? Hell, no, because in addition to being memory-unsafe, porting an FFI is *much* harder than porting a bit of C support code.

But porting the FFI is /not/ harder than porting a JIT (believe me), and as an FFI grows in sophistication it gets easier to port because techniques and solutions used on one platform are applicable to others (again, I've been there and done that either with discarding the bulk of the assembler in an existing implementation or a new port in the VisualWorks FFI implementations on Alpha AXP, MIPS, PowerPC, SPARC (32-bit & 64-bit) and x86-64).

It really is a matter of where you want to deal with certain aspects

...

of the system. I prefer to deal with that level of stuff in a plugin via C because:

• it provides a way to do ensure memory safety

there's nothing in plugins per-se to maintain memory safety. A mistake in a plugin is as destructive as a mistake in an FFI call. In either case a wrapper around a particular call can validate and provide safety.

In theory, yes. In practice such wrappers *do* exist in plugins and they do *not* exist in FFI calls. Why? Good question. Probably because people understand that in C land there are more constraints and they are mentally "set up" to keep an eye on the constraints and to make sure the incoming stuff is okay.

I don't agree that they don't exist in Smalltalk code. Well-written Smalltalk code that uses an FFI typically does provide an abstraction-boundary and manage memory behind it. The ODBC connect is a good example.

+ it provides a way to provide abstractions

...

• so does Smalltalk, and t least in SMalltalk we have inheritance to

enable us to express cross-platform APIs in abstract classes and map these down onto particular platforms in concrete subclasses. This is /far better/ than the sprawling mess that is the platforms tree in the VM.

In theory yes. In practice, who knows. Having looked at some Smalltalks that expose all the platform details (like Dolphin) it seems to me that the equation of "platform FFI code * number of platforms" will be /far more/ of a sprawling mess. Which is probably why (unless I'm mistaken in which case I'll appreciate a correction) precisely none of the cross-platform Smalltalks do that.

Two great advantages that Smalltalk has here are browsability and inheritance. It is straight-forward to use inheritance to group platform fan-out and the IDE supports comparing the various platform-specific implementations. Further, inheritance encourages code-sharing and keeping platform fan-out to a minimum. In contrast the platform tree in the Squeak VM has very little shared code and IME is quite badly organized.

+ the code is faster to write and debug in C

...

this is debatable.

Hardly. Let's start with "#include <windows.h>" (or whatever platform stuff you happen to need). You're losing right there. The problem is that the system isn't set up to seamlessly interact with C. Even if you've got some auto-magic header file importer that creates structs and defines and stuff, as long as it takes more than, i.e.,

winMessageBox: contents title: title <include: 'windows.h'> MessageBox(NULL, contents, title, MB_OK + MB_ICONWARNING).

you're going to lose because it takes so much longer to get to the point of actually writing the code you're trying to write.

That's a strong point. It was hard in VisualWorks to find the effort to keep the C preprocessor up-to-date. However, I can at least imagine a well-maintained interface to windows.h and know how I would implement it (I've discussed this a lot before; I won't bore you).

Secondly, you *really* need to try the latest Visual Studio and play with its debugger - I was blown away while debugging SqueakSSL when I hit a typical NULL pointer, fixed the code in my plugin DLL, continued in VS and the underlying Squeak KEPT RUNNING! Holy crap. This stuff has come a looooong way.

True. But it breaks my heart. I don't want to be embraced and then extended ;) They're playing catch-up and have probably caught up, but I would still like to compete.

In the VW FFI with the ability to catch exceptions

...

and pass them back up (doesn't always work, a bad emory corruption may crash the entire system) makes thing easier. But low-level debugging is painful in general.

• access to facilities (threads, interrupts, atomicity) not

available otherwise

Again, a strong FFI provides at least some of these facilities. Not everything can be done through the FFI, but a lot can be done, and elegantly and extensibly, by Smalltalk programmers, not a few VM specialists.

I call BS. The reality is that one needs to know three kinds of things, only one of which would be helped by using the FFI:

1. The "mechanics" of writing a plugin, i.e., slang and compilation. This

is the part where the FFI would help, but so would better APIs for marshaling. While it's "cool" that one can run slang plugins in Squeak it is also completely and utterly *useless* when it comes to integrating platform specific stuff and having to deal with Slang, VMMaker and the awful build setups, and then -on top of that- the actual work you're trying to do is just a bit too much.

2. Understanding the rules of interfacing the external world, i.e., when

can pointers be kept, what does GC do to your objects etc. None of this is helped by using the FFI.

3. Understanding the actual domain code. Again, none of this is helped by

the FFI (to the contrary because nobody will find examples for the weird Smalltalk selectors we use on the web and people will in turn not know where to find the actual value for MB_OK or somesuch).

Basically, the assumption that *all* the difficulty is in writing the marshaling code is nonsense. There is some of it, true, but that is entirely our fault for providing poor integration APIs.

Agreed.

On the downside, there is:

...

• it's harder to modify for people who don't know C and are not set

up to build a plugin

• it's often aimed at the smallest common denominator (though not

necessarily)

• it is a far more fixed interface that is far harder to evolve

Yes, but when you call it "stable" instead of "fixed" and "robust" instead of "harder to evolve" it becomes a plus :-)

There's plenty of examples of stable interfaces above the FFI. VisualWorks has depended on its database connects; they are its bread if not its butter and are implemented on the FFI. The Windows registry is accessed solely through the FFI, etc. These interfaces are certainly stable and have evolved.

best Eliot

Cheers,

• Andreas

  Cheers,

...

 - Andreas


   On the other hand, a VM providing a larger de facto immutable API:
   - somehow is more secure (a guaranty we can run very old images in
   very new OSes).
   - can maintain the illusion that despite the efforts of OS and
   hardware designers to make it ever more complex, a single person

can still understand (almost) the whole system. - avoid the necessity to embed knowledge of dozens of different variants of different OSes in your image.

   Of course, the complexity still exists under the carpet... When I
   contemplate all the unecessary complex knowledge from those
   beautiful
   configure/cmake scripts and macros, I'm not convinced all this

cruft will be easier to modify in Smalltalk than it is in C world.

   I perfectly understand too the desire of a VM maintainer to lighten
   the burden ;) though.

   Nicolas

   2010/9/29 Eliot Miranda<eliot.miranda@gmail.com
   <mailto:eliot.miranda@gmail.com>>:





       On Wed, Sep 29, 2010 at 10:22 AM, Andreas
       Raab<andreas.raab@gmx.de <mailto:andreas.raab@gmx.de>>  wrote:



           On 9/29/2010 9:36 AM, Eliot Miranda wrote:


                   See the recordMouseEvent() and its friends
               (sqNextEventPut,
                   eventBuffer[1024] ...) in sqWin32Window, for
               better understanding my
                   point :)


               Which makes my point well.  These are simply
               maintaining a queue.  If
               one has callbacks then the bulk of the Windows event
               handling system can
               be lifted up into the image.  This is what Vassili
               Bykov did in Newspeak
               above my Alien callbacks.  In Newspeak the Windows
               MainWndProc is a
               callback:


           Thus losing any level of platform abstraction. A bad
           trade-off, IMO.


       The platform abstraction merely moves from the VM to classes
       in the image.  But it does mean that one can properly
       implement Windows event semantics (things like query quit
       events) that the queue prevents.  And it means that the VM
       gets simpler and complexity is handled by better facilities
       (in Smalltalk with its many advantages for creating and
       maintaining abstractions with concrete variant

implementations). But you and I have had this discussion a number of times and I take your point about the VM maintaining a black-box abstraction boundary whereas the in-image implementation is in theory porous. As I've said before this doesn't strike me as that important when Smalltalk has had boundary violators such as instVarAt:[put:] for ever and they are rarely abused. If one were to implement a native GUI event interface in Smalltalk using callbacks one would have to police the abstraction boundary. But that's easily done, and there are other benefits - the VM, being simpler, gets more longevity since changing the event interface doesn't imply a VM change - the system, having native interfaces, can support each platform's facilities in full instead of providing the lowest common denominator So IMO it is a good trade-off. best Eliot

           Cheers,
             - Andreas

On 29 September 2010 22:40, Bert Freudenberg bert@freudenbergs.de wrote:

On 29.09.2010, at 20:53, Nicolas Cellier wrote:

...

When I contemplate all the unecessary complex knowledge from those beautiful configure/cmake scripts and macros, I'm not convinced all this cruft will be easier to modify in Smalltalk than it is in C world.

You still have to know C and the C libraries to do anything at that level. Using FFI does not buy you anything in this regard.

While other languages see themselves as C extensions / glue languages and happily expose C constructs directly, IMHO Squeak should not depend on FFI, but use proper VM abstractions. Supporting FFI is good, depending on it, not.

My intent is to make VM flexible. Sure thing, VM should provide a default OS-specific layer. The pity is, that i can't alter/replace it, when using FFI.

• Bert -