Hi Craig,
first, the message from the non-MT VM, "Warning; callback failed to own the VM", indicates that a callback is coming in on other than the VM thread. Here's the non-MT implementation from sqVirtualMachine.c:
sqInt ownVM(sqInt threadIdAndFlags) { extern sqInt amInVMThread(void); return amInVMThread() ? 0 : -1; }
So with the normal VM any callbacks must come in on the thread that made a callout.
On Tue, Oct 6, 2015 at 7:23 AM, Craig Latta craig@netjam.org wrote:
Hm, I still get the "failed to own the VM" message with themultithreaded VM. I thought it would wait until it could run the callback from a VM thread? What now?
Second, the MT VM /is/ only a prototype. But second, what's the platform, etc? Do you have a reproducible case? Are you willing to use gdb et al to debug ownVM?
The issue here is when a callback comes in from some unknown thread, what do we do with it? By default it is rejected. But if there is a process in the foreignCallbackProcessSlot of the specialObjectsArray, it will be cloned to handle each "foreign" callback. Likely you don't have such a process installed, hence the callbacks getting rejected.
If you read CogInterpreterMT>>ownVM: you'll see
CogInterpreterMT>>ownVM: threadIndexAndFlags <api> <inline: false> "This is the entry-point for plugins and primitives that wish to reacquire the VM after having released it via disownVM or callbacks that want to acquire it without knowing their ownership status. This call will block until the VM is owned by the current thread or an error occurs. The argument should be the value answered by disownVM, or 0 for callbacks that don't know if they have disowned or not. This is both an optimization to avoid having to query thread- local storage for the current thread's index (since it can easily keep it in some local variable), and a record of when an unbound process becomes affined to a thread for the dynamic extent of some operation.
Answer 0 if the current thread is known to the VM. Answer 1 if the current thread is unknown to the VM and takes ownership. Answer -1 if the current thread is unknown to the VM and fails to take ownership." | threadIndex flags vmThread myProc activeProc sched | <var: #vmThread type: #'CogVMThread *'> threadIndexAndFlags = 0 ifTrue: [^self ownVMFromUnidentifiedThread]. ...
and CogInterpreterMT>>ownVMFromUnidentifiedThread "Attempt to take ownership from a thread that as yet doesn't know its index. This supports callbacks where the callback could originate from any thread. Answer 0 if the owning thread is known to the VM. Answer 1 if the owning thread is unknown to the VM and now owns the VM. Answer -1 if the owning thread is unknown to the VM and fails to own the VM. Answer -2 if the owning thread is unknown to the VM and there is no foreign callback process installed." | count threadIndex vmThread | <var: #vmThread type: #'CogVMThread *'> <inline: false> (threadIndex := cogThreadManager ioGetThreadLocalThreadIndex) ~= 0 ifTrue: [ "this is a callback from a known thread" threadIndex = cogThreadManager getVMOwner ifTrue: "the VM has not been disowned" [self assert: (disowningVMThread isNil or: [disowningVMThread = self currentVMThread]). disowningVMThread := nil. self currentVMThread state: CTMAssignableOrInVM. ^VMAlreadyOwnedHenceDoNotDisown]. ^self ownVM: threadIndex]. foreignCallbackPriority = 0 ifTrue: [^-2]. ...
i.e. that to accept a callback from an unknown thread the system has to have the priority at which to run callbacks from unknown threads (foreignCallbackPriority) determined, and that's set by the priority of the process filling the foreignCallbackProcessSlot in the specialObjectsArray, as set by disownVM:.
CogInterpreterMT>>disownVM: flags "Release the VM to other threads and answer the current thread's index. Currently valid flags: DisownVMLockOutFullGC - prevent fullGCs while this thread disowns the VM. OwnVMForeignThreadFlag - indicates lowest-level entry from a foreign thread - not to be used explicitly by clients - only set by ownVMFromUnidentifiedThread VMAlreadyOwnedHenceDoNotDisown - indicates an ownVM from a callback was made when the vm was still owned. - not to be used explicitly by clients - only set by ownVMFromUnidentifiedThread
This is the entry-point for plugins and primitives that wish to release the VM while performing some operation that may potentially block, and for callbacks returning back to some blocking operation. If this thread does not reclaim the VM before- hand then when the next heartbeat occurs the thread manager will schedule a thread to acquire the VM which may start running the VM in place of this thread.
N.B. Most of the state needed to resume after preemption is set in preemptDisowningThread." <api> <inline: false> | vmThread result | <var: #vmThread type: #'CogVMThread *'> ... (flags anyMask: DisownVMForProcessorRelinquish) ifTrue: [| proc | (proc := objectMemory splObj: foreignCallbackProcessSlot) ~= objectMemory nilObject ifTrue: [foreignCallbackPriority := self quickFetchInteger: PriorityIndex ofObject: proc]. relinquishing := true. self sqLowLevelMFence].
So how to install a process in the foreignCallbackProcessSlot? See SmalltalkImage>>recreateSpecialObjectsArray.
SmalltalkImage>>recreateSpecialObjectsArray "Smalltalk recreateSpecialObjectsArray" "To external package developers: **** DO NOT OVERRIDE THIS METHOD. ***** If you are writing a plugin and need additional special object(s) for your own use, use addGCRoot() function and use own, separate special objects registry " "The Special Objects Array is an array of objects used by the Squeak virtual machine. Its contents are critical and accesses to it by the VM are unchecked, so don't even think of playing here unless you know what you are doing." | newArray | newArray := Array new: 60. "Nil false and true get used throughout the interpreter" newArray at: 1 put: nil. newArray at: 2 put: false. newArray at: 3 put: true. ... "Used to be WeakFinalizationList for WeakFinalizationList hasNewFinalization, obsoleted by ephemeron support." newArray at: 56 put: nil.
"reserved for foreign callback process"
newArray at: 57 put: (self specialObjectsArray at: 57 ifAbsent: []).
newArray at: 58 put: #unusedBytecode. "59 reserved for Sista counter tripped message" newArray at: 59 put: #conditionalBranchCounterTrippedOn:. "60 reserved for Sista class trap message" newArray at: 60 put: #classTrapFor:.
"Now replace the interpreter's reference in one atomic operation" self specialObjectsArray becomeForward: newArray
So chose a priority to run foreign callbacks at, write a pair of accessors to set and get the foreign callback process in the specialObjectsArray. Set the slot to a new process ([] newProcess priority: N; yourself), save and restart the image (to allow the system to initialize correctly), and then see how you get on. I expect you'll see crashes soon enough because...the MT VM is a prototype. But you may just get lucky. I certainly hope so!
Ah yes, you also have to arrange that when that process runs it replaces itself in the foreignCallbackSlot, so you'll have to work out a scheme to handle multiple foreign callbacks. I wrote a scheme for VW, but haven't got this far with the Cog MT VM.
_,,,^..^,,,_ best, Eliot