Does anyone know whether it's possible to create a Form backed by an out-of-line (e.g. mmaped) Bitmap-like object?
Tony
Hi Tony,
On Jun 23, 2022, at 7:25 AM, Tony Garnock-Jones tonyg@leastfixedpoint.com wrote:
Does anyone know whether it's possible to create a Form backed by an out-of-line (e.g. mmaped) Bitmap-like object?
It isn’t implemented in any way yet but we have wanted to do something like this for a while, e.g. providing memory mapped byte arrays for intra-image communication. It should be possible to implement given pinning. However there are detailed issues, foremost being where does the mapping have to start relative to the object? (obviously the mapping starts on a vm page boundary). Must the first field of the object be the start of the mapping or can it include the header of the object? The latter is far more convenient for the implementation.
Can you say more about your use case?
Tony
Eliot _,,,^..^,,,_ (phone)
On 6/23/22 18:15, Eliot Miranda wrote:
Can you say more about your use case?
I've been implementing the X11 protocol in pure Smalltalk and I want to use a little bit of FFI to shmget/shmat and use that to get the pixels over to the X server. At present XPutImage is working but flooding the server with XPutImage seems to run into a rate limit from time to time so it's not very usable. Best to switch to mmap/shm if I can.
Tony
On Thu, Jun 23, 2022 at 10:28 AM Tony Garnock-Jones < tonyg@leastfixedpoint.com> wrote:
On 6/23/22 18:15, Eliot Miranda wrote:
Can you say more about your use case?
I've been implementing the X11 protocol in pure Smalltalk and I want to use a little bit of FFI to shmget/shmat and use that to get the pixels over to the X server. At present XPutImage is working but flooding the server with XPutImage seems to run into a rate limit from time to time so it's not very usable. Best to switch to mmap/shm if I can.
Technical:
So in this use including the header in the first page would be ok? Would the mapping be read-only from the perspective of the X server? Do you need shmget to be run via the FFI or would it be OK to invoke it implicitly as part of a mapping allocation primitive? Would it be better to create the mapping via the FFI and provide the mapping to the allocation primitive? How should the mapping be taken down?
Windows folks, how portable is this idea to other OS's, especially windows? (I'm guessing this is a standard facility on most leading OS's). So would it be worth evolving towards a cross-platform abstraction?
Procedural: Would you be prepared to build the VM to test this? Perhaps even generate it from sources? To what extent would you be able to collaborate with e.g. Tom & myself in implementing this?
_,,,^..^,,,_ best, Eliot
On 6/23/22 19:41, Eliot Miranda wrote:
So in this use including the header in the first page would be ok?
Yes, this would be fine. Ideally the beginning of the data portion would be page-aligned but I suspect the server may not actually care, and anything 8- or 16-byte aligned would probably be just fine.
Would the mapping be read-only from the perspective of the X server?
It could be, yes, but ideally it'd be possible to have it read-write so that I could grab pixels from the server as well as send them.
The address of the mapping may have to be fixed for the lifetime of the relationship with the server - I don't know why, but the address is included in the wire protocol, not just the segment identifier. I need to dig deeper.
Do you need shmget to be run via the FFI or would it be OK to invoke it implicitly as part of a mapping allocation primitive?
It'd be OK if it were done primitively but I do need access to the shm ID etc.
mmap incidentally can work as well but I need to implement fd passing which is fiddly and involves sendmsg. I'm putting it off if I can :-)
Would it be better to create the mapping via the FFI and provide the mapping to the allocation primitive?
It might be better if the primitives could cope with chunks of external memory from arbitrary sources - then I could manage shm and/or mmap myself via the FFI.
How should the mapping be taken down?
By manual invocation of some kind of free()-analogue, I suspect; so from the GC's point of view, the object holding the external bytevector is permanent until explicitly, manually released. And I think it's probably best if the image-side is responsible for deallocating the chunk of memory too (obvs if it is supplied in a mechanism-agnostic way to the GC this holds double!)
Windows folks, how portable is this idea to other OS's, especially windows? (I'm guessing this is a standard facility on most leading OS's). So would it be worth evolving towards a cross-platform abstraction?
If you get to hand chunks-o-RAM to the GC without having to talk about how you got them, it might be pretty portable automatically.
Procedural: Would you be prepared to build the VM to test this? Perhaps even generate it from sources? To what extent would you be able to collaborate with e.g. Tom & myself in implementing this?
Yes, I would be happy building. I would also be happy generating from sources, though I might have silly questions about the process for the first couple of hours. I'd be happy to collaborate.
Cheers, Tony
On 6/23/22 20:16, Tony Garnock-Jones wrote:
How should the mapping be taken down?
There's another interesting question here, namely what happens to externally-allocated chunks across image-save/restore. I think FFI has a general facility for managing external resources that could work?
Hi,
Am Do., 23. Juni 2022 um 19:42 Uhr schrieb Eliot Miranda < eliot.miranda@gmail.com>:
On Thu, Jun 23, 2022 at 10:28 AM Tony Garnock-Jones < tonyg@leastfixedpoint.com> wrote:
On 6/23/22 18:15, Eliot Miranda wrote:
Can you say more about your use case?
I've been implementing the X11 protocol in pure Smalltalk and I want to use a little bit of FFI to shmget/shmat and use that to get the pixels over to the X server. At present XPutImage is working but flooding the server with XPutImage seems to run into a rate limit from time to time so it's not very usable. Best to switch to mmap/shm if I can.
Technical:
So in this use including the header in the first page would be ok? Would the mapping be read-only from the perspective of the X server? Do you need shmget to be run via the FFI or would it be OK to invoke it implicitly as part of a mapping allocation primitive? Would it be better to create the mapping via the FFI and provide the mapping to the allocation primitive? How should the mapping be taken down?
Windows folks, how portable is this idea to other OS's, especially windows? (I'm guessing this is a standard facility on most leading OS's). So would it be worth evolving towards a cross-platform abstraction?
If the question is "does Windows support shared memory in a way that is similar to POSIX shmget/shmat", I can definitely say yes. At work we have software that runs on both Windows and AIX, uses shared memory for inter-process communication under the hood, and has the API calls for both platforms abstracted uniformly for both platforms.
https://docs.microsoft.com/en-us/windows/win32/memory/sharing-files-and-memo... shmget --> CreateFileMapping (or OpenFileMapping if you never want to create at the call site) shmat --> MapViewOfFile (or MapViewOfFileEx) shmdt --> UnmapViewOfFile shmctl(IPC_RMID, ...) --> CloseHandle...
Of course there are different flags and security attributes for which the portability will vary.
Moreover, if you want to wrap OS shared memory objects in Squeak, also consider doing the same for OS semaphores or other synchronization primitives. If two processes want to write, or you don't want to read inconsistent data in the absence of atomic writes, you will want some kind of inter-process synchronization.
I cannot comment on the applicability for a Windows X Server.
Kind regards, Jakob
squeak-dev@lists.squeakfoundation.org
-
Eliot Miranda -
Jakob Reschke -
Tony Garnock-Jones