On 20 Aug 2015, at 21:28, vm-dev-request@lists.squeakfoundation.org wrote:
Message: 1 Date: Thu, 20 Aug 2015 17:51:43 +0200 From: Esteban Lorenzano <estebanlm@gmail.com mailto:estebanlm@gmail.com> Subject: [Vm-dev] nested structures with FFI? To: Squeak Virtual Machine Development Discussion <vm-dev@lists.squeakfoundation.org mailto:vm-dev@lists.squeakfoundation.org> Message-ID: <4F31ED4B-6246-43D6-B85B-7215D8570D28@gmail.com mailto:4F31ED4B-6246-43D6-B85B-7215D8570D28@gmail.com> Content-Type: text/plain; charset=utf-8
Hi,
I’m doing some tests with nested structures (to handle the conversion of NativeBoost-FFI to regular FFI) and I’m having some problems when accessing fields in the nested structure. For example, this tests:
FFITestNestingStructure class>>#fieldsDesc ^ #( byte one; FFITestStructure nested; )
FFITestStructure class>>#fieldsDesc ^ #( byte byte; short short; long long; float float; double double; int64 int64; )
(definition is NB-compatible, so do not take the differences with regular FFI into account)
testNestedStructure | s1 |
s1 := FFITestNestingStructure new. s1 nested byte: 42. self assert: s1 nested byte = 42
testExternallyAllocatedNestedStructure | s1 |
s1 := FFITestNestingStructure externalNew. self assert: s1 getHandle class = ExternalAddress. s1 nested byte: 42. self assert: s1 nested byte = 42
fails always because
s1 nested byte = 0.
(which is obviously bad).
I remember Eliot saying that structures with FFI was not being optimal, and I wonder (before start digging more in deep) if this could have something to do?
I use nested structs in NB for the libgit2 bindings and they work as advertised. There can be problems if some structure is packed (i.e. uses non-4-byte alignment for some field) but that shouldn’t apply in your test case.
Not sure if this helps you in any way but I thought I’d let you know :)
Cheers, Max
cheers, Esteban
On 20 Aug 2015, at 21:48, Max Leske maxleske@gmail.com wrote:
On 20 Aug 2015, at 21:28, vm-dev-request@lists.squeakfoundation.org mailto:vm-dev-request@lists.squeakfoundation.org wrote:
Message: 1 Date: Thu, 20 Aug 2015 17:51:43 +0200 From: Esteban Lorenzano <estebanlm@gmail.com mailto:estebanlm@gmail.com> Subject: [Vm-dev] nested structures with FFI? To: Squeak Virtual Machine Development Discussion <vm-dev@lists.squeakfoundation.org mailto:vm-dev@lists.squeakfoundation.org> Message-ID: <4F31ED4B-6246-43D6-B85B-7215D8570D28@gmail.com mailto:4F31ED4B-6246-43D6-B85B-7215D8570D28@gmail.com> Content-Type: text/plain; charset=utf-8
Hi,
I’m doing some tests with nested structures (to handle the conversion of NativeBoost-FFI to regular FFI) and I’m having some problems when accessing fields in the nested structure. For example, this tests:
FFITestNestingStructure class>>#fieldsDesc ^ #( byte one; FFITestStructure nested; )
FFITestStructure class>>#fieldsDesc ^ #( byte byte; short short; long long; float float; double double; int64 int64; )
(definition is NB-compatible, so do not take the differences with regular FFI into account)
testNestedStructure | s1 |
s1 := FFITestNestingStructure new. s1 nested byte: 42. self assert: s1 nested byte = 42
testExternallyAllocatedNestedStructure | s1 |
s1 := FFITestNestingStructure externalNew. self assert: s1 getHandle class = ExternalAddress. s1 nested byte: 42. self assert: s1 nested byte = 42
fails always because
s1 nested byte = 0.
(which is obviously bad).
I remember Eliot saying that structures with FFI was not being optimal, and I wonder (before start digging more in deep) if this could have something to do?
I use nested structs in NB for the libgit2 bindings and they work as advertised. There can be problems if some structure is packed (i.e. uses non-4-byte alignment for some field) but that shouldn’t apply in your test case.
Not sure if this helps you in any way but I thought I’d let you know :)
I do not think so :) I’m quite sure this is not going to be a problem when using the ThreadedFFI backend instead ASMJIT… now my problem is with this backend, for the moment :)
Esteban
Cheers, Max
cheers, Esteban
On 20 Aug 2015, at 22:47, Esteban Lorenzano estebanlm@gmail.com wrote:
On 20 Aug 2015, at 21:48, Max Leske <maxleske@gmail.com mailto:maxleske@gmail.com> wrote:
On 20 Aug 2015, at 21:28, vm-dev-request@lists.squeakfoundation.org mailto:vm-dev-request@lists.squeakfoundation.org wrote:
Message: 1 Date: Thu, 20 Aug 2015 17:51:43 +0200 From: Esteban Lorenzano <estebanlm@gmail.com mailto:estebanlm@gmail.com> Subject: [Vm-dev] nested structures with FFI? To: Squeak Virtual Machine Development Discussion <vm-dev@lists.squeakfoundation.org mailto:vm-dev@lists.squeakfoundation.org> Message-ID: <4F31ED4B-6246-43D6-B85B-7215D8570D28@gmail.com mailto:4F31ED4B-6246-43D6-B85B-7215D8570D28@gmail.com> Content-Type: text/plain; charset=utf-8
Hi,
I’m doing some tests with nested structures (to handle the conversion of NativeBoost-FFI to regular FFI) and I’m having some problems when accessing fields in the nested structure. For example, this tests:
FFITestNestingStructure class>>#fieldsDesc ^ #( byte one; FFITestStructure nested; )
FFITestStructure class>>#fieldsDesc ^ #( byte byte; short short; long long; float float; double double; int64 int64; )
(definition is NB-compatible, so do not take the differences with regular FFI into account)
testNestedStructure | s1 |
s1 := FFITestNestingStructure new. s1 nested byte: 42. self assert: s1 nested byte = 42
testExternallyAllocatedNestedStructure | s1 |
s1 := FFITestNestingStructure externalNew. self assert: s1 getHandle class = ExternalAddress. s1 nested byte: 42. self assert: s1 nested byte = 42
fails always because
s1 nested byte = 0.
(which is obviously bad).
I remember Eliot saying that structures with FFI was not being optimal, and I wonder (before start digging more in deep) if this could have something to do?
I use nested structs in NB for the libgit2 bindings and they work as advertised. There can be problems if some structure is packed (i.e. uses non-4-byte alignment for some field) but that shouldn’t apply in your test case.
Not sure if this helps you in any way but I thought I’d let you know :)
I do not think so :) I’m quite sure this is not going to be a problem when using the ThreadedFFI backend instead ASMJIT… now my problem is with this backend, for the moment :)
I’m just realised that this behaviour is correct according current ExternalStructure implementation, because:
nested "This method was automatically generated" ^FFITestStructure fromHandle: (handle structAt: 2 length: 27)
and
ByteArray>>#structAt: byteOffset length: length "Return a structure of the given length starting at the indicated byte offset." | value | value := ByteArray new: length. 1 to: length do:[:i| value unsignedByteAt: i put: (self unsignedByteAt: byteOffset+i-1)]. ^value
So it is giving me always a copy… So we need a better solution (a non copy way to access)? I’m thinking on passing a “MappedByteArray” something that can work with the same original ByteArray (or ExternalAddress, is the same), to affect the referenced data… what do you think?
cheers! Esteban
Esteban
Cheers, Max
cheers, Esteban
On 21 Aug 2015, at 04:02, Esteban Lorenzano estebanlm@gmail.com wrote:
On 20 Aug 2015, at 22:47, Esteban Lorenzano <estebanlm@gmail.com mailto:estebanlm@gmail.com> wrote:
On 20 Aug 2015, at 21:48, Max Leske <maxleske@gmail.com mailto:maxleske@gmail.com> wrote:
On 20 Aug 2015, at 21:28, vm-dev-request@lists.squeakfoundation.org mailto:vm-dev-request@lists.squeakfoundation.org wrote:
Message: 1 Date: Thu, 20 Aug 2015 17:51:43 +0200 From: Esteban Lorenzano <estebanlm@gmail.com mailto:estebanlm@gmail.com> Subject: [Vm-dev] nested structures with FFI? To: Squeak Virtual Machine Development Discussion <vm-dev@lists.squeakfoundation.org mailto:vm-dev@lists.squeakfoundation.org> Message-ID: <4F31ED4B-6246-43D6-B85B-7215D8570D28@gmail.com mailto:4F31ED4B-6246-43D6-B85B-7215D8570D28@gmail.com> Content-Type: text/plain; charset=utf-8
Hi,
I’m doing some tests with nested structures (to handle the conversion of NativeBoost-FFI to regular FFI) and I’m having some problems when accessing fields in the nested structure. For example, this tests:
FFITestNestingStructure class>>#fieldsDesc ^ #( byte one; FFITestStructure nested; )
FFITestStructure class>>#fieldsDesc ^ #( byte byte; short short; long long; float float; double double; int64 int64; )
(definition is NB-compatible, so do not take the differences with regular FFI into account)
testNestedStructure | s1 |
s1 := FFITestNestingStructure new. s1 nested byte: 42. self assert: s1 nested byte = 42
testExternallyAllocatedNestedStructure | s1 |
s1 := FFITestNestingStructure externalNew. self assert: s1 getHandle class = ExternalAddress. s1 nested byte: 42. self assert: s1 nested byte = 42
fails always because
s1 nested byte = 0.
(which is obviously bad).
I remember Eliot saying that structures with FFI was not being optimal, and I wonder (before start digging more in deep) if this could have something to do?
I use nested structs in NB for the libgit2 bindings and they work as advertised. There can be problems if some structure is packed (i.e. uses non-4-byte alignment for some field) but that shouldn’t apply in your test case.
Not sure if this helps you in any way but I thought I’d let you know :)
I do not think so :) I’m quite sure this is not going to be a problem when using the ThreadedFFI backend instead ASMJIT… now my problem is with this backend, for the moment :)
I’m just realised that this behaviour is correct according current ExternalStructure implementation, because:
nested "This method was automatically generated" ^FFITestStructure fromHandle: (handle structAt: 2 length: 27)
and
ByteArray>>#structAt: byteOffset length: length "Return a structure of the given length starting at the indicated byte offset." | value | value := ByteArray new: length. 1 to: length do:[:i| value unsignedByteAt: i put: (self unsignedByteAt: byteOffset+i-1)]. ^value
So it is giving me always a copy… So we need a better solution (a non copy way to access)? I’m thinking on passing a “MappedByteArray” something that can work with the same original ByteArray (or ExternalAddress, is the same), to affect the referenced data… what do you think?
So yes, introducing a "referenced handle” (who applies offsets to same memory address or byte array) makes everything work… and looks like the correct way to handle this. Do you want I commit this changes to FFI package or I keep them in my own stuff?
Esteban
cheers! Esteban
Esteban
Cheers, Max
cheers, Esteban
2015-08-21 5:57 GMT+02:00 Esteban Lorenzano estebanlm@gmail.com:
On 21 Aug 2015, at 04:02, Esteban Lorenzano estebanlm@gmail.com wrote:
On 20 Aug 2015, at 22:47, Esteban Lorenzano estebanlm@gmail.com wrote:
On 20 Aug 2015, at 21:48, Max Leske maxleske@gmail.com wrote:
On 20 Aug 2015, at 21:28, vm-dev-request@lists.squeakfoundation.org wrote:
Message: 1 Date: Thu, 20 Aug 2015 17:51:43 +0200 From: Esteban Lorenzano estebanlm@gmail.com Subject: [Vm-dev] nested structures with FFI? To: Squeak Virtual Machine Development Discussion vm-dev@lists.squeakfoundation.org Message-ID: 4F31ED4B-6246-43D6-B85B-7215D8570D28@gmail.com Content-Type: text/plain; charset=utf-8
Hi,
I’m doing some tests with nested structures (to handle the conversion of NativeBoost-FFI to regular FFI) and I’m having some problems when accessing fields in the nested structure. For example, this tests:
FFITestNestingStructure class>>#fieldsDesc ^ #( byte one; FFITestStructure nested; )
FFITestStructure class>>#fieldsDesc ^ #( byte byte; short short; long long; float float; double double; int64 int64; )
(definition is NB-compatible, so do not take the differences with regular FFI into account)
testNestedStructure | s1 |
s1 := FFITestNestingStructure new. s1 nested byte: 42. self assert: s1 nested byte = 42
testExternallyAllocatedNestedStructure | s1 |
s1 := FFITestNestingStructure externalNew. self assert: s1 getHandle class = ExternalAddress. s1 nested byte: 42. self assert: s1 nested byte = 42
fails always because
s1 nested byte = 0.
(which is obviously bad).
I remember Eliot saying that structures with FFI was not being optimal, and I wonder (before start digging more in deep) if this could have something to do?
I use nested structs in NB for the libgit2 bindings and they work as advertised. There can be problems if some structure is packed (i.e. uses non-4-byte alignment for some field) but that shouldn’t apply in your test case.
Not sure if this helps you in any way but I thought I’d let you know :)
I do not think so :) I’m quite sure this is not going to be a problem when using the ThreadedFFI backend instead ASMJIT… now my problem is with this backend, for the moment :)
I’m just realised that this behaviour is correct according current ExternalStructure implementation, because:
nested "This method was automatically generated" ^FFITestStructure fromHandle: (handle structAt: 2 length: 27)
and
ByteArray>>#structAt: byteOffset length: length "Return a structure of the given length starting at the indicated byte offset." | value | value := ByteArray new: length. 1 to: length do:[:i| value unsignedByteAt: i put: (self unsignedByteAt: byteOffset+i-1)]. ^value
So it is giving me always a copy… So we need a better solution (a non copy way to access)? I’m thinking on passing a “MappedByteArray” something that can work with the same original ByteArray (or ExternalAddress, is the same), to affect the referenced data… what do you think?
So yes, introducing a "referenced handle” (who applies offsets to same memory address or byte array) makes everything work… and looks like the correct way to handle this. Do you want I commit this changes to FFI package or I keep them in my own stuff?
Esteban
Yes!
years ago when I ported Smallapack to Squeak, that just was the missing sugar that would have enabled implementing all using Smalltalk memory...
Instead of that, I had to copy data to external heap in order to perform certain operations. For example, for extracting the imaginary part of a complex matrix interlaced representation { re[0] , im[0] , re[1] , im[1], ... } one trick is to cast a DoubleComplexMatrix * into a DoubleMatrix *, and start the copy at doubleMatrixPointer+1, with increment=2...
Nicolas
cheers! Esteban
Esteban
Cheers, Max
cheers, Esteban
On 21 Aug 2015, at 11:45, Nicolas Cellier nicolas.cellier.aka.nice@gmail.com wrote:
2015-08-21 5:57 GMT+02:00 Esteban Lorenzano <estebanlm@gmail.com mailto:estebanlm@gmail.com>:
On 21 Aug 2015, at 04:02, Esteban Lorenzano <estebanlm@gmail.com mailto:estebanlm@gmail.com> wrote:
On 20 Aug 2015, at 22:47, Esteban Lorenzano <estebanlm@gmail.com mailto:estebanlm@gmail.com> wrote:
On 20 Aug 2015, at 21:48, Max Leske <maxleske@gmail.com mailto:maxleske@gmail.com> wrote:
On 20 Aug 2015, at 21:28, vm-dev-request@lists.squeakfoundation.org mailto:vm-dev-request@lists.squeakfoundation.org wrote:
Message: 1 Date: Thu, 20 Aug 2015 17:51:43 +0200 From: Esteban Lorenzano <estebanlm@gmail.com mailto:estebanlm@gmail.com> Subject: [Vm-dev] nested structures with FFI? To: Squeak Virtual Machine Development Discussion <vm-dev@lists.squeakfoundation.org mailto:vm-dev@lists.squeakfoundation.org> Message-ID: <4F31ED4B-6246-43D6-B85B-7215D8570D28@gmail.com mailto:4F31ED4B-6246-43D6-B85B-7215D8570D28@gmail.com> Content-Type: text/plain; charset=utf-8
Hi,
I’m doing some tests with nested structures (to handle the conversion of NativeBoost-FFI to regular FFI) and I’m having some problems when accessing fields in the nested structure. For example, this tests:
FFITestNestingStructure class>>#fieldsDesc ^ #( byte one; FFITestStructure nested; )
FFITestStructure class>>#fieldsDesc ^ #( byte byte; short short; long long; float float; double double; int64 int64; )
(definition is NB-compatible, so do not take the differences with regular FFI into account)
testNestedStructure | s1 |
s1 := FFITestNestingStructure new. s1 nested byte: 42. self assert: s1 nested byte = 42
testExternallyAllocatedNestedStructure | s1 |
s1 := FFITestNestingStructure externalNew. self assert: s1 getHandle class = ExternalAddress. s1 nested byte: 42. self assert: s1 nested byte = 42
fails always because
s1 nested byte = 0.
(which is obviously bad).
I remember Eliot saying that structures with FFI was not being optimal, and I wonder (before start digging more in deep) if this could have something to do?
I use nested structs in NB for the libgit2 bindings and they work as advertised. There can be problems if some structure is packed (i.e. uses non-4-byte alignment for some field) but that shouldn’t apply in your test case.
Not sure if this helps you in any way but I thought I’d let you know :)
I do not think so :) I’m quite sure this is not going to be a problem when using the ThreadedFFI backend instead ASMJIT… now my problem is with this backend, for the moment :)
I’m just realised that this behaviour is correct according current ExternalStructure implementation, because:
nested "This method was automatically generated" ^FFITestStructure fromHandle: (handle structAt: 2 length: 27)
and
ByteArray>>#structAt: byteOffset length: length "Return a structure of the given length starting at the indicated byte offset." | value | value := ByteArray new: length. 1 to: length do:[:i| value unsignedByteAt: i put: (self unsignedByteAt: byteOffset+i-1)]. ^value
So it is giving me always a copy… So we need a better solution (a non copy way to access)? I’m thinking on passing a “MappedByteArray” something that can work with the same original ByteArray (or ExternalAddress, is the same), to affect the referenced data… what do you think?
So yes, introducing a "referenced handle” (who applies offsets to same memory address or byte array) makes everything work… and looks like the correct way to handle this. Do you want I commit this changes to FFI package or I keep them in my own stuff?
Esteban
Yes!
years ago when I ported Smallapack to Squeak, that just was the missing sugar that would have enabled implementing all using Smalltalk memory...
Instead of that, I had to copy data to external heap in order to perform certain operations. For example, for extracting the imaginary part of a complex matrix interlaced representation { re[0] , im[0] , re[1] , im[1], ... } one trick is to cast a DoubleComplexMatrix * into a DoubleMatrix *, and start the copy at doubleMatrixPointer+1, with increment=2…
yes, but you are talking of something more complex than what I did, I think (what I did just works for nested structures)… what you want is a functionality I also believe I will need to add anyway :) so I will keep playing with this and will commit when ready.
cheers! Esteban
Nicolas
cheers! Esteban
Esteban
Cheers, Max
cheers, Esteban
2015-08-21 11:53 GMT+02:00 Esteban Lorenzano estebanlm@gmail.com:
On 21 Aug 2015, at 11:45, Nicolas Cellier < nicolas.cellier.aka.nice@gmail.com> wrote:
2015-08-21 5:57 GMT+02:00 Esteban Lorenzano estebanlm@gmail.com:
On 21 Aug 2015, at 04:02, Esteban Lorenzano estebanlm@gmail.com wrote:
On 20 Aug 2015, at 22:47, Esteban Lorenzano estebanlm@gmail.com wrote:
On 20 Aug 2015, at 21:48, Max Leske maxleske@gmail.com wrote:
On 20 Aug 2015, at 21:28, vm-dev-request@lists.squeakfoundation.org wrote:
Message: 1 Date: Thu, 20 Aug 2015 17:51:43 +0200 From: Esteban Lorenzano estebanlm@gmail.com Subject: [Vm-dev] nested structures with FFI? To: Squeak Virtual Machine Development Discussion vm-dev@lists.squeakfoundation.org Message-ID: 4F31ED4B-6246-43D6-B85B-7215D8570D28@gmail.com Content-Type: text/plain; charset=utf-8
Hi,
I’m doing some tests with nested structures (to handle the conversion of NativeBoost-FFI to regular FFI) and I’m having some problems when accessing fields in the nested structure. For example, this tests:
FFITestNestingStructure class>>#fieldsDesc ^ #( byte one; FFITestStructure nested; )
FFITestStructure class>>#fieldsDesc ^ #( byte byte; short short; long long; float float; double double; int64 int64; )
(definition is NB-compatible, so do not take the differences with regular FFI into account)
testNestedStructure | s1 |
s1 := FFITestNestingStructure new. s1 nested byte: 42. self assert: s1 nested byte = 42
testExternallyAllocatedNestedStructure | s1 |
s1 := FFITestNestingStructure externalNew. self assert: s1 getHandle class = ExternalAddress. s1 nested byte: 42. self assert: s1 nested byte = 42
fails always because
s1 nested byte = 0.
(which is obviously bad).
I remember Eliot saying that structures with FFI was not being optimal, and I wonder (before start digging more in deep) if this could have something to do?
I use nested structs in NB for the libgit2 bindings and they work as advertised. There can be problems if some structure is packed (i.e. uses non-4-byte alignment for some field) but that shouldn’t apply in your test case.
Not sure if this helps you in any way but I thought I’d let you know :)
I do not think so :) I’m quite sure this is not going to be a problem when using the ThreadedFFI backend instead ASMJIT… now my problem is with this backend, for the moment :)
I’m just realised that this behaviour is correct according current ExternalStructure implementation, because:
nested "This method was automatically generated" ^FFITestStructure fromHandle: (handle structAt: 2 length: 27)
and
ByteArray>>#structAt: byteOffset length: length "Return a structure of the given length starting at the indicated byte offset." | value | value := ByteArray new: length. 1 to: length do:[:i| value unsignedByteAt: i put: (self unsignedByteAt: byteOffset+i-1)]. ^value
So it is giving me always a copy… So we need a better solution (a non copy way to access)? I’m thinking on passing a “MappedByteArray” something that can work with the same original ByteArray (or ExternalAddress, is the same), to affect the referenced data… what do you think?
So yes, introducing a "referenced handle” (who applies offsets to same memory address or byte array) makes everything work… and looks like the correct way to handle this. Do you want I commit this changes to FFI package or I keep them in my own stuff?
Esteban
Yes!
years ago when I ported Smallapack to Squeak, that just was the missing sugar that would have enabled implementing all using Smalltalk memory...
Instead of that, I had to copy data to external heap in order to perform certain operations. For example, for extracting the imaginary part of a complex matrix interlaced representation { re[0] , im[0] , re[1] , im[1], ... } one trick is to cast a DoubleComplexMatrix * into a DoubleMatrix *, and start the copy at doubleMatrixPointer+1, with increment=2…
yes, but you are talking of something more complex than what I did, I think (what I did just works for nested structures)… what you want is a functionality I also believe I will need to add anyway :) so I will keep playing with this and will commit when ready.
cheers! Esteban
If we have possibility to pass a composed object (ByteArray + byte_offset) to FFI and have it transformed in a (char *) ByteArray_start_of_data + byte_offset, then it solves all the problems. That's the only feature I need, the complex part in my example (casting etc...) is completely handled in image code.
Nicolas
cheers! Esteban
Esteban
Cheers, Max
cheers, Esteban
On 21 Aug 2015, at 12:42, Nicolas Cellier nicolas.cellier.aka.nice@gmail.com wrote:
2015-08-21 11:53 GMT+02:00 Esteban Lorenzano <estebanlm@gmail.com mailto:estebanlm@gmail.com>:
On 21 Aug 2015, at 11:45, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com mailto:nicolas.cellier.aka.nice@gmail.com> wrote:
2015-08-21 5:57 GMT+02:00 Esteban Lorenzano <estebanlm@gmail.com mailto:estebanlm@gmail.com>:
On 21 Aug 2015, at 04:02, Esteban Lorenzano <estebanlm@gmail.com mailto:estebanlm@gmail.com> wrote:
On 20 Aug 2015, at 22:47, Esteban Lorenzano <estebanlm@gmail.com mailto:estebanlm@gmail.com> wrote:
On 20 Aug 2015, at 21:48, Max Leske <maxleske@gmail.com mailto:maxleske@gmail.com> wrote:
On 20 Aug 2015, at 21:28, vm-dev-request@lists.squeakfoundation.org mailto:vm-dev-request@lists.squeakfoundation.org wrote:
Message: 1 Date: Thu, 20 Aug 2015 17:51:43 +0200 From: Esteban Lorenzano <estebanlm@gmail.com mailto:estebanlm@gmail.com> Subject: [Vm-dev] nested structures with FFI? To: Squeak Virtual Machine Development Discussion <vm-dev@lists.squeakfoundation.org mailto:vm-dev@lists.squeakfoundation.org> Message-ID: <4F31ED4B-6246-43D6-B85B-7215D8570D28@gmail.com mailto:4F31ED4B-6246-43D6-B85B-7215D8570D28@gmail.com> Content-Type: text/plain; charset=utf-8
Hi,
I’m doing some tests with nested structures (to handle the conversion of NativeBoost-FFI to regular FFI) and I’m having some problems when accessing fields in the nested structure. For example, this tests:
FFITestNestingStructure class>>#fieldsDesc ^ #( byte one; FFITestStructure nested; )
FFITestStructure class>>#fieldsDesc ^ #( byte byte; short short; long long; float float; double double; int64 int64; )
(definition is NB-compatible, so do not take the differences with regular FFI into account)
testNestedStructure | s1 |
s1 := FFITestNestingStructure new. s1 nested byte: 42. self assert: s1 nested byte = 42
testExternallyAllocatedNestedStructure | s1 |
s1 := FFITestNestingStructure externalNew. self assert: s1 getHandle class = ExternalAddress. s1 nested byte: 42. self assert: s1 nested byte = 42
fails always because
s1 nested byte = 0.
(which is obviously bad).
I remember Eliot saying that structures with FFI was not being optimal, and I wonder (before start digging more in deep) if this could have something to do?
I use nested structs in NB for the libgit2 bindings and they work as advertised. There can be problems if some structure is packed (i.e. uses non-4-byte alignment for some field) but that shouldn’t apply in your test case.
Not sure if this helps you in any way but I thought I’d let you know :)
I do not think so :) I’m quite sure this is not going to be a problem when using the ThreadedFFI backend instead ASMJIT… now my problem is with this backend, for the moment :)
I’m just realised that this behaviour is correct according current ExternalStructure implementation, because:
nested "This method was automatically generated" ^FFITestStructure fromHandle: (handle structAt: 2 length: 27)
and
ByteArray>>#structAt: byteOffset length: length "Return a structure of the given length starting at the indicated byte offset." | value | value := ByteArray new: length. 1 to: length do:[:i| value unsignedByteAt: i put: (self unsignedByteAt: byteOffset+i-1)]. ^value
So it is giving me always a copy… So we need a better solution (a non copy way to access)? I’m thinking on passing a “MappedByteArray” something that can work with the same original ByteArray (or ExternalAddress, is the same), to affect the referenced data… what do you think?
So yes, introducing a "referenced handle” (who applies offsets to same memory address or byte array) makes everything work… and looks like the correct way to handle this. Do you want I commit this changes to FFI package or I keep them in my own stuff?
Esteban
Yes!
years ago when I ported Smallapack to Squeak, that just was the missing sugar that would have enabled implementing all using Smalltalk memory...
Instead of that, I had to copy data to external heap in order to perform certain operations. For example, for extracting the imaginary part of a complex matrix interlaced representation { re[0] , im[0] , re[1] , im[1], ... } one trick is to cast a DoubleComplexMatrix * into a DoubleMatrix *, and start the copy at doubleMatrixPointer+1, with increment=2…
yes, but you are talking of something more complex than what I did, I think (what I did just works for nested structures)… what you want is a functionality I also believe I will need to add anyway :) so I will keep playing with this and will commit when ready.
cheers! Esteban
If we have possibility to pass a composed object (ByteArray + byte_offset) to FFI and have it transformed in a (char *) ByteArray_start_of_data + byte_offset, then it solves all the problems. That's the only feature I need, the complex part in my example (casting etc...) is completely handled in image code.
yes, I understood that… and we still do not have it in FFI. But with Igor we talked the other day about it, and he needed to implement it in NativeBoost to speed up display rendering (when replacing VM display with SDL display). So I will need to implement it as an FFI extension too… but I’m still not there (close, but not there). What I implemented for now is just a way to avoid structure copying when accessing nested structures… something that was making the access style “a.b.c = 42” fail :)
Esteban
Nicolas
cheers! Esteban
Esteban
Cheers, Max
cheers, Esteban
vm-dev@lists.squeakfoundation.org
-
Esteban Lorenzano -
Max Leske -
Nicolas Cellier