Hi everyone,

I am of particularly low expertise in these matters, but in the grand tradition of online discourse I thought I'd chime in anyway.

Tim Rowledge wrote on Sun, 28 Jun 2020 17:53:21 -0700

...

Seriously. We've been sitting around twiddling thumbs about better

graphics for decades now.

...

We had Balloon2D & 3D. We had 'Pooh graphics' to do vector forms. We have some excellent stuff being done by the Cuis crew. We have some amazing ideas from Nile/Gezira. We have a number of potential 3rd party graphics libraries we might

adopt.

...

We have an advanced JIT that could be used to make on-demand graphics

routines either on their own or working with pretty much any of the above.

...

We probably have other options I've not even heard of.

Not sure if everyone keeps up with the guys on the other side of the tracks, but the Bloc library in Pharo took the approach of using Skia https://skia.org/ (originally it used Moz2D). A lot of work went into it, and a year or two ago I was able to get some interesting things up in my own experiments. This work is now no longer being done on the mainline branch, but instead as part of the Feenk GToolkit project. I'm not sure if or when the Pharo people will pick it up for incorporation.

When talking about incorporating 3rd party libraries, do we mean as VM plugins or through FFI? Is there some advantage of one over the other? On an unrelated thread (in which I posted off-topic, shame) I mentioned that Vulkan provides a complete XML description of its API https://github.com/KhronosGroup/Vulkan-Docs/blob/master/xml/vk.xml, which, as far as I'm able to understand these things, means the enormous pain of creating FFI bindings is somewhat ameliorated here. That said, I don't really know anything about Vulkan other than the fact it's supposed to be the "next generation" graphics api from the same producers that brought us *OpenGL* and *OpenGL 2: The Reckoning*.

On Sun, Jun 28, 2020 at 11:01 PM Jecel Assumpcao Jr jecel@merlintec.com

wrote:

Nile/Gezira + JIT would be my vote, but it has been planned for a long time and, as far as I know, is not happening right now.

Nile seems like a fascinating option and I've tried to read as much about it as I can. Unfortunately aside from the various posted talks online and the VPRI papers, there's not a lot of information about using it (unless I am missing something -- you can track a github issue about its status here https://github.com/damelang/nile/issues/3). My guess is that the Gezira part, of which there is even less information, is very closely tied to the specific Frank environment in which it was used -- is that correct? Using plain Squeak how would one even incorporate Nile?

Though the survey I did for the Board pointed to OpenGL ES as our best

option (at the time, things have changed since), it is not something I personally like. I want something that is not a black box so I can single step from the highest level to individual pixels when debugging issues. But I also want it to take advantage of all available resources (including GPUs, extra processors) by translating debugged code into something else. I just don't want to debug in that something else, which would be the case for OpenGL ES shaders.

One thing about Squeak and friends that is very nice is the way it can abstract over essential "kinds" of activity in the computer. For example, the global `Processor` class manages everything I really need from my actual computer's processor: scheduling different things to run and then doing so, while providing me a no-nonsense interface into itself. The current BitBlt graphics stack is also a sweet-spot abstraction for the kinds of graphics it's working with. Once you dig down and get the hang of bits, forms, and canvases, it all comes together in a delightfully lively way. I see what Jecel is saying here and I think it's a worthy goal -- what would be the equivalent of the `Processor` global for the GPU (even in cases where the actual machine doesn't have a "real" GPU)?

Is GPU& shader  support necessary/desirable: yes if real-time rendering for thighs like games/VR are important, but in that case there is a bigger problem jitter caused by GC,

In the image I just uploaded two days ago and that one can get from http://www.zogotounga.net/comp/squeak/rogue.htm the main feature is a 2D real-time top-down game.

I found that the GC-caused jitter was indeed a problem, but also that the main culprit was the time it takes to simply paint an image which is a surprisingly (to me) slow operation.

That, plus the alpha-blending issue that was discussed on the VM list a little ago, where Juan proposed two new primitives.

Stef

On Fri, Jul 03, 2020 at 06:15:17PM +0200, Hilaire wrote:

Le 30/06/2020 ?? 08:26, St??phane Rollandin a ??crit??:

...

...

Is GPU& shader ??support necessary/desirable: yes if real-time rendering for thighs??like games/VR are important, but in that case there is a bigger problem jitter caused by GC,

In the image I just uploaded two days ago and that one can get from http://www.zogotounga.net/comp/squeak/rogue.htm the main feature is a 2D real-time top-down game.

I found that the GC-caused jitter was indeed a problem, but also that the main culprit was the time it takes to simply paint an image which is a surprisingly (to me) slow operation.

That, plus the alpha-blending issue that was discussed on the VM list a little ago, where Juan proposed two new primitives.

Juan proposed these primitives to speed up the rendering of VectorGraphics Widget Morph.

I remember a discussion about that, though I don't have a link to it. It sounds like we need to follow up on this and make something happen.

Regarding graphics, Juan's work on Vector Graphics can be of interest for Squeak.

Definitely yes. Juan is working towards *both* high quality rendering *and* high performance. I think that there may be a perception that Morphic 3 and VectorGraphics might sacrifice performance for rendering quality, but I am not sure that is the case at all.

This recent post by Juan would be of interest to any of us attempting to achieve graphics performance on modern hardware:

https://lists.cuis.st/mailman/archives/cuis-dev/2020-June/001958.html

I also have to add that the recent work that you (Hilaire) have been doing with DrGeo on Cuis really illustrates the potential. Your circular menu will be of interest to anyone who has been involved with Morphic, Etoys, and Scratch interfaces:

https://mamot.fr/@drgeo/104310151994863180

Dave

On 2020-07-03, at 1:32 PM, Hilaire hilaire@drgeo.eu wrote:

Le 03/07/2020 à 22:21, David T. Lewis a écrit :

...

...

Juan proposed these primitives to speed up the rendering of VectorGraphics Widget Morph.

https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/505

I'm all for such improvements but I have no idea about updating the rather specialised ARM bitblt code that Ben Avison did for RPF a few years ago. That stuff is responsible for no small part of the really good UI performance we get on Pi.

tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim When all else fails, let a = 7. If that doesn't help, then read the manual.

What are we going to actually try to do here? Who has time/interest/skill?

Yes, Juan's suggested bitblt improvements sound good. Indeed pretty much anything he's done would be. This all has the advantage of being 'in system' too, with maybe some VM work to get the best results.

Connecting to outside libraries always has costs; after all they're not exactly likely to be set up just as we'd like. An advantage of a decent open source library is that we could at least consider hacking at it to make a version that talks nicely to our bitmaps instead of of messing around with copy/convert to get the results back. More work though... Another option *might* be to use the algorithms if we can't really use the code. Still more work.

We have a framework for dynamic translation in the cog vms. We could use much of it to do dynamically translated bitblts. The ARM specialised bitblt code kinda-sorta does that but we could do much better I think. Maybe that would make borrowing algorithms more sensible than linking to an outside library?

I'm not a huge fan of a direct connect to openGL because of the issues like Apple not supporting it any more, and which version can a Raspberry Pi run etc. That might make sometihng like Skia more sensible since it seems to be aimed at multiple back ends and so maybe we could benefit indirectly. Of course, it leaves us vulnerable to Error 33.

The Nile/Gezira stuff obviously sounds great but there is the question of actually getting it working usefully, let alone finishing it. Will that ever happen? Is what exists good enough and complete enough to be of use?

There is a downloadable 'Frank' image that seems to start up ok at http://tinlizzie.org/~ohshima/Frank-170908.zip It is a Squeak 4.4 image as far as I can tell so you need an older VM of roughly that vintage - pre-Spur anyway.

The small examples mostly work ok (see things like GeziraCanvas class>example* & GZKernel class>example*) though one or two take a very long time.

tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim State-of-the-art: What we could do with enough money.

Changing the subject line to focus on a specific proposal by Juan Vuletich for an enhancement to BitBlt rules.

The idea originated from discussion on the Cuis list, and the proposal is documented in an opensmalltalk-vm issue here:

https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/505

While I am generally familiar with VM issues, I have no experience with BitBlt and I am not sure exactly what needs to be done to make this happen. There are VM (plugin) changes needed, and some image side support will needed as well (at least for fallback code in the image).

So - who knows how to make this happen, and how do we make it so?

Thanks, Dave

On Fri, Jul 03, 2020 at 10:32:17PM +0200, Hilaire wrote:

Le 03/07/2020 ?? 22:21, David T. Lewis a ??crit??:

...

...

Juan proposed these primitives to speed up the rendering of VectorGraphics Widget Morph.

I remember a discussion about that, though I don't have a link to it. It sounds like we need to follow up on this and make something happen.

Here it is:

https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/505

-- Dr. Geo http://drgeo.eu https://pouet.chapril.org/@hilaire

On Mon, Jun 29, 2020 at 12:01:07AM -0300, Jecel Assumpcao Jr wrote:

Tim Rowledge wrote on Sun, 28 Jun 2020 17:53:21 -0700

...

Seriously. We've been sitting around twiddling thumbs about better graphics for decades now.

We had Balloon2D & 3D. We had 'Pooh graphics' to do vector forms. We have some excellent stuff being done by the Cuis crew. We have some amazing ideas from Nile/Gezira. We have a number of potential 3rd party graphics libraries we might adopt. We have an advanced JIT that could be used to make on-demand graphics routines either on their own or working with pretty much any of the above. We probably have other options I've not even heard of.

You didn't mention the original BitBlt and WarpBlit, which is understandable if your focus is on vector graphics.

...

Maybe we should actually do something? Can we at least talk about it?

Though the survey I did for the Board pointed to OpenGL ES as our best option (at the time, things have changed since), it is not something I personally like. I want something that is not a black box so I can single step from the highest level to individual pixels when debugging issues. But I also want it to take advantage of all available resources (including GPUs, extra processors) by translating debugged code into something else. I just don't want to debug in that something else, which would be the case for OpenGL ES shaders.

Nile/Gezira + JIT would be my vote, but it has been planned for a long time and, as far as I know, is not happening right now.

-- Jecel

During the 2017 time frame, when Jecel was serving on the Squeak Oversight Board, we had a number of conversations on this topic and Jacel provided some welcome insight and direction. I looked through our notes from that time, and excerpted the following from our SOB minutes on Google docs:

Squeak board meeting 2017-04-19: --------------------------------

Mo’ better graphics; so what about adopting Nile etc? (Bert says unfinished and unlikely to get finished. OpenGL might be an option but tricky to use. Juan’s Morphic3 looks fab but is nowhere near complete so far as we can tell.)

Squeak board meeting 2017-06-07: --------------------------------

Jecel researched OpenCL. Here are 3 plans based on that: 1) Juan is using FFI to talk to OpenCL, and "kernels" are just Smalltalk strings 2) we could extend Slang to have OpenCL functionality. Then it would be possible to debug (very sloooowly) in the normal Squeak. The .cl kernel code would then be generated from Slang 3) Nile (or equivalent) could be used instead of Slang and would be translated to Squeak for debugging and later to OpenCL for performance OpenGL could be an alternative to OpenCL and would work on the Raspberry Pi

Squeak board meeting 2017-08-16: --------------------------------

Many Squeaky 3D-graphics implementations… - OpenGL - CroquetGL (checked out on Pi, does the OpenGL>example at 150fps) - MorphicGL - Athens (large, complex, system to be able to use GL, Cairo, bitblt, anything) - Rome ( Cairo base, used in Sophie project?) - WebGL / three.js / A-Frame via SqueakJS - Terf stuff

Graphics conclusions, draft based on Dave's (mis)understanding: For future work in 3D graphics, the preferred direction is OpenGL-ES.

(Did we rule out Vulkan yet? -- Ah, bad for 2D. Nevermind, then. Apple doesn't plan to support Vulkan since it has Metal instead)

Existing Squeak GL interfaces (generally Croquet based) are based on older versions of OpenGL and are not our preferred direction.

We do not currently have an implementation of OpenGL-ES for Squeak, and no people or funding are currently dedicated to the problem. Funding and/or volunteer efforts would be welcome.

Though the survey I did for the Board pointed to OpenGL ES as our best
option (at the time, things have changed since), it is not something I
personally like. I want something that is not a black box so I can
single step from the highest level to individual pixels when debugging
issues.

I have the same  thoughts on this. Faster, smoother graphics would be really good. At the same time I like access to everything within the image.

+1

Stef

If you could only choose one of these, which would you choose?

If new modern, faster, but opaque graphics system is implemented, then it does not have to replace the existing, fully accessible one, it can just complement it (being available via a specific Canvas subclass, and able to deliver a Form on demand). There is no need to have an opposition here. Just as we have Morphic *and* MVC, we can have BitBlt *and* a bridge to an opaque backend for people interested in that kind of performance.

As for me, I am interested in both.

Stef

Hi, I have tried to use the OpenGL from Squeak and found it very opaque and hard to use. Both because the Squeak side is not very well documented and because OpenGL is hard itself.

I have also looked briefly at Dolphin Smalltalk which have quite fast graphics on Windows but the way you get it to work is quite elaborate and convoluted.

I agree with what Stephane Rollandin wrote; keep BitBlt for simplicity and work on better way to access whatever we choose to use as a fast backend.

Best, Karl

On Fri, Jul 3, 2020 at 5:11 PM Peter Crowther peter@ozzard.org wrote:

On Thu, 2 Jul 2020 at 09:22, karl ramberg karlramberg@gmail.com wrote:

...

Faster, smoother graphics would be really good. At the same time I like access to everything within the image.

If you could only choose one of these, which would you choose? "Access to

everything within the image" is a really large anchor to attach to a graphics subsystem these days. We have a single-threaded Smalltalk that has deep assumptions about a single memory space and a single rectangular display area of consistent properties wired in over forty years of development. Meanwhile, modern graphics subsystems are *very* heavily parallel, use custom hardware, and you try to offload everything you can to the coprocessor because you don't want it even in your memory space.

If we want a teaching and learning system that can be understood and modified by a normal programmer, we keep everything in the image and accept limited facilities and slowness. If we want something that makes use of all of the varied and wonderful capabilities of modern graphics hardware then a) what's the smallest subset we'll accept (to make it easy to port Squeak to new platforms), b) what knowledge do we require of anyone who wishes to investigate the graphics subsystem?

Cheers,

• Peter

To put it bluntly: I gave porting old-school MVC to Cuis.

[flame on] Morphic is miserable and really needs to be replaced by MVC, though (IMHO).

Why?

1. There’s a reason views and controllers were initially separated; that reason is reusability and better factoring.

2. The composite design pattern is a good thing (https://en.wikipedia.org/wiki/Composite_pattern https://en.wikipedia.org/wiki/Composite_pattern) and should be applied to everything where one builds trees, such as view hierarchies. The root view object should not have subviews, the view-composite should. (I know that old-school MVC didn’t do it that way, but new implementations certainly should).

3. In a good OO framework, the more abstract classes generally have the minimum necessary state (or none at all). Class Morph has 7 instance variables.

4. The goal of OO design is to reuse specifications across multiple implementations, so why does class Morph rename the usual geometry methods of bounds, extent, width, height, etc. as (e.g.,) morphBounds?

I could go on… [flame off]

It would be really nice to have a simple MVC (and display-list graphics) framework that could be placed on top of several graphics back-ends including BitBLT, OpenGL and HTML5.

stp

--

Stephen Travis Pope Santa Barbara, California, USA http://HeavenEverywhere.com http://heaveneverywhere.com/ http://FASTLabInc.com http://fastlabinc.com/ https://vimeo.com/user19434036/videos https://vimeo.com/user19434036/videos http://heaveneverywhere.com/Reflections

--

On Jul 9, 2020, at 11:12 PM, stephen@heaveneverywhere.com wrote:

…on a related topic, I’ve started porting old-school MVC* to Cuis with the intention of replacing Morphic with a generic framework that can be back-ended by BitBLT, OpenGL or HTML5.

Anybody want to help?

Stephen

• = starting with traditional MVC classes and system windows, and following that with some version of the “Wrapper” framework as still used in VisualWorks.

--

                Stephen Travis Pope   Santa Barbara, California, USA    

<pastedGraphic.tiff> http://HeavenEverywhere.com http://heaveneverywhere.com/ http://FASTLabInc.com http://fastlabinc.com/ https://vimeo.com/user19434036/videos https://vimeo.com/user19434036/videos http://heaveneverywhere.com/Reflections http://heaveneverywhere.com/Reflections

--

...

On Jun 28, 2020, at 5:53 PM, tim Rowledge <tim@rowledge.org mailto:tim@rowledge.org> wrote:

Seriously. We've been sitting around twiddling thumbs about better graphics for decades now.

We had Balloon2D & 3D. We had 'Pooh graphics' to do vector forms. We have some excellent stuff being done by the Cuis crew. We have some amazing ideas from Nile/Gezira. We have a number of potential 3rd party graphics libraries we might adopt. We have an advanced JIT that could be used to make on-demand graphics routines either on their own or working with pretty much any of the above. We probably have other options I've not even heard of.

Maybe we should actually do something? Can we at least talk about it?

tim

tim Rowledge; tim@rowledge.org mailto:tim@rowledge.org; http://www.rowledge.org/tim http://www.rowledge.org/tim Strange OpCodes: SVE: Skip on Vernal Equinox

Hi Stephen,

1. There’s a reason views and controllers were initially separated; that reason is reusability and better factoring.

Afaik a good equivalent for that separation in Morphic are morph + model. If you are going to develop a sophisticated UI, you can use the ToolBuilder framework, and your tool will be even decoupled from the UI framework.

2. The composite design pattern is a good thing (https://en.wikipedia.org/wiki/Composite_pattern) and should be applied to everything where one builds trees, such as view hierarchies.

Morphic uses the composite pattern, too, doesn't it? All morphs are nested in each other, beginning at the root morph (the world), and form a big object tree, just like MVC does. Please correct me if I am wrong.

3. In a good OO framework, the more abstract classes generally have the minimum necessary state (or none at all). Class Morph has 7 instance variables.

And class View has 13 instance variables. :-) I agree with you that abstract class should not have any unnecessary state, but in fact, graphics aren't a completely abstract domain. A minimum of geometry information (bounds) is vital for graphic representation, and some node links (submorphs, owner) are vital, too, for navigating through the Morphic tree. Note that Morph even has a cleaned-up state by extracting all non-crucial fields into the extension object.

4. The goal of OO design is to reuse specifications across multiple implementations, so why does class Morph rename the usual geometry methods of bounds, extent, width, height, etc. as (e.g.,) morphBounds?

I'm afraid I don't get your point here. My image does not even know the selector #morphBounds. But #extent, #width, and #height, for example, are derived attributes, which means that an implementor only needs to manipulate the single underlying representation, which is bounds, in order to adjust all specified attributes. Are you desiring something like an abstract morph interface that does not yet define core and derived attributes? It's true that Morph is not an abstract class (Morph new openInWorld works like a charm), but in which practical example would you need this?

It would be really nice to have a simple MVC (and display-list graphics) framework that could be placed on top of several graphics back-ends including BitBLT, OpenGL and HTML5.

That's an interesting thought! As said above, Morphic is a composite framework, too, so couldn't you start by implementing #printHtmlOn: on Morph, next to #drawOn:? But in practice, I guess your desire would be very hard to achieve because HTML5 is a high-level OO framework while OpenGL operates on a quite low level. HTML5 is kind of a competing object tree against Morphic, so you could either try to keep two trees in sync or rebuild one of the trees periodically, which would be performance-extensive. For OpenGL, couldn't something like an OpenGLCanvas be a starting point?

Best, Christoph

________________________________ Von: Squeak-dev squeak-dev-bounces@lists.squeakfoundation.org im Auftrag von Stephen Pope stephen@heaveneverywhere.com Gesendet: Samstag, 11. Juli 2020 20:02:37 An: The general-purpose Squeak developers list; Discussion of Cuis Smalltalk Betreff: Re: [squeak-dev] We need to talk about graphics

To put it bluntly: I gave porting old-school MVC to Cuis.

[flame on] Morphic is miserable and really needs to be replaced by MVC, though (IMHO).

Why?

1. There’s a reason views and controllers were initially separated; that reason is reusability and better factoring.

2. The composite design pattern is a good thing (https://en.wikipedia.org/wiki/Composite_pattern) and should be applied to everything where one builds trees, such as view hierarchies. The root view object should not have subviews, the view-composite should. (I know that old-school MVC didn’t do it that way, but new implementations certainly should).

3. In a good OO framework, the more abstract classes generally have the minimum necessary state (or none at all). Class Morph has 7 instance variables.

4. The goal of OO design is to reuse specifications across multiple implementations, so why does class Morph rename the usual geometry methods of bounds, extent, width, height, etc. as (e.g.,) morphBounds?

I could go on… [flame off]

It would be really nice to have a simple MVC (and display-list graphics) framework that could be placed on top of several graphics back-ends including BitBLT, OpenGL and HTML5.

stp

--

Stephen Travis Pope Santa Barbara, California, USA http://HeavenEverywhere.com http://FASTLabInc.com https://vimeo.com/user19434036/videos http://heaveneverywhere.com/Reflections

--

On Jul 9, 2020, at 11:12 PM, stephen@heaveneverywhere.commailto:stephen@heaveneverywhere.com wrote:

…on a related topic, I’ve started porting old-school MVC* to Cuis with the intention of replacing Morphic with a generic framework that can be back-ended by BitBLT, OpenGL or HTML5.

Anybody want to help?

Stephen

* = starting with traditional MVC classes and system windows, and following that with some version of the “Wrapper” framework as still used in VisualWorks.

--

Stephen Travis Pope Santa Barbara, California, USA <pastedGraphic.tiff> http://HeavenEverywhere.comhttp://heaveneverywhere.com/ http://FASTLabInc.comhttp://fastlabinc.com/ https://vimeo.com/user19434036/videos http://heaveneverywhere.com/Reflections

--

On Jun 28, 2020, at 5:53 PM, tim Rowledge <tim@rowledge.orgmailto:tim@rowledge.org> wrote:

Seriously. We've been sitting around twiddling thumbs about better graphics for decades now.

We had Balloon2D & 3D. We had 'Pooh graphics' to do vector forms. We have some excellent stuff being done by the Cuis crew. We have some amazing ideas from Nile/Gezira. We have a number of potential 3rd party graphics libraries we might adopt. We have an advanced JIT that could be used to make on-demand graphics routines either on their own or working with pretty much any of the above. We probably have other options I've not even heard of.

Maybe we should actually do something? Can we at least talk about it?

tim -- tim Rowledge; tim@rowledge.orgmailto:tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: SVE: Skip on Vernal Equinox

Playing with stuff; tried a WatchMorph to look at circles drawn by Balloon. Grabbing one via World->new morph->from alphabetical list produces a decently round clockface with a coloured centre section. However, using the debug menu to turn on the anti-aliasing results in a rather ugly lumpiness, as if it were a fruit that has been poked a bit too often to test for ripeness. Weird. The B2Dplugin appears to be active and I can make it do profiling etc that makes it pretty certain it is being used. Is this an ARM/Pi issue... nope, looks the same on a Mac system.

tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Gotta run, the cat's caught in the printer.