On Fri, Jan 6, 2017 at 6:33 PM Eliot Miranda eliot.miranda@gmail.com wrote:

Hi Guille,

...

On Jan 6, 2017, at 6:44 AM, Guillermo Polito guillermopolito@gmail.com

wrote:

...

Hi,

I was checking the code in sqUnixHeartbeat.c to see how the heartbeat

thread/itimer worked. It somehow bothers me that there are different compiled artifacts, one per option.

...

What do you think about having a VM that manages that as an argument

provided when we launch the VM? This would add some flexibility that we don't have right now because we make the decision at compile time.

I think it's a fine idea but it isn't really the issue. The issue is that the itimer mechanism is problematic, especially for foreign code, and is therefore a stop gap. The itimer interrupts long-running system calls, which means that things like sound libraries break (at Qwaq I had to fix ALSA to get it to work with the itimer heartbeat). Since Pharo is becoming more reliant on external code it may impact us more going forward.

The real issue is that linux's requirement that thread priorities be set in per-application file in /etc/security/limits.d (IIRC) is a big. Neither Windows nor Mac OS X requires such nonsense, and a threaded heartbeat is used on those systems without any issue at all. Why linux erected this mess in the first place is something I don't understand.

I had to implement the itimer heartbeat to get Qwaq forums running on Linux running pre 2.6 kernels, but had many other problems to solve as a result (ALSA, database connects).

Were it that the vm merely had to detect whether it could use the threaded heartbeat then things would be easy. Instead one can only use the thing if one has superuser permissions to install a file in /etc, just to use a thread of higher priority than the main one.

Thanks for the explanation, Eliot. I had no idea how bad the issues are with the itimer, but I'm glad you also see the user-facing issue with the heartbeat.

An alternative might be to lower the priority of the main thread. Then the file installation would be unnecessary.

Could you elaborate a little bit more on this idea? How could this impact the vm? What could be the drawbacks here?

Fabio

To summarize, the itimer heartbeat is to be avoided as much as possible. It causes hard to debug issues with external code, has to be turned off and on around fork. It's a stop gap. Having to install a file in /etc just to be able to use a thread is insane (and AFAICT unique to linux). Whatever you do in the short term to deal with these problems I'll support, but in the long term we simply want a threaded heartbeat without needing to install anything.

...

The code in sqUnixHeartbeat.c is not a lot nor very complex, it should

not be difficult to do...

...

Also, what would be the drawbacks besides an increase on the vm size?

I hope I've explained above that I expect the drawbacks will be intermittent failures of external code.

...

Guille

Hi Fabio, Hi Guille,

On Fri, Jan 6, 2017 at 9:44 AM, Fabio Niephaus lists@fniephaus.com wrote:

On Fri, Jan 6, 2017 at 6:33 PM Eliot Miranda eliot.miranda@gmail.com wrote:

...

Hi Guille,

...

On Jan 6, 2017, at 6:44 AM, Guillermo Polito guillermopolito@gmail.com

wrote:

...

Hi,

I was checking the code in sqUnixHeartbeat.c to see how the heartbeat

thread/itimer worked. It somehow bothers me that there are different compiled artifacts, one per option.

...

What do you think about having a VM that manages that as an argument

provided when we launch the VM? This would add some flexibility that we don't have right now because we make the decision at compile time.

I think it's a fine idea but it isn't really the issue. The issue is that the itimer mechanism is problematic, especially for foreign code, and is therefore a stop gap. The itimer interrupts long-running system calls, which means that things like sound libraries break (at Qwaq I had to fix ALSA to get it to work with the itimer heartbeat). Since Pharo is becoming more reliant on external code it may impact us more going forward.

The real issue is that linux's requirement that thread priorities be set in per-application file in /etc/security/limits.d (IIRC) is a big. Neither Windows nor Mac OS X requires such nonsense, and a threaded heartbeat is used on those systems without any issue at all. Why linux erected this mess in the first place is something I don't understand.

I had to implement the itimer heartbeat to get Qwaq forums running on Linux running pre 2.6 kernels, but had many other problems to solve as a result (ALSA, database connects).

Were it that the vm merely had to detect whether it could use the threaded heartbeat then things would be easy. Instead one can only use the thing if one has superuser permissions to install a file in /etc, just to use a thread of higher priority than the main one.

Thanks for the explanation, Eliot. I had no idea how bad the issues are with the itimer, but I'm glad you also see the user-facing issue with the heartbeat.

...

An alternative might be to lower the priority of the main thread. Then the file installation would be unnecessary.

Could you elaborate a little bit more on this idea? How could this impact the vm? What could be the drawbacks here?

First of all, for the heartbeat thread to work reliably it must run at higher priority than the thread running Smalltalk code. This is because its job is to cause Smalltalk code to break out at regular intervals to check for events. If the Smalltalk code is compute-intensive then it will prevent the heartbeat thread from running unless the heartbeat thread is running at a higher priority, and so it will be impossible to receive input keys, etc. (Note that if event collection was in a separate thread it would suffer the same issue; compute intensive code would block the event collection thread unless it was running at higher priority).

Right now, Linux restricts creating threads with priority higher than the default to those programs that have a /etc/security/limits.d/program.conf file that specifies the highest priority thread the program can create. And prior to the 2.6.12 kernel only superuser processes could create higher-priority threads. I do know that prior to 2.6.12 one couldn't create threads of *lower* priority than the default either (I would have used this if I could).

If 2.6.12 allows a program to create threads with lower priorities *without* needing a /etc/security/limits.d/program.conf, or more conveniently to allow a thread's priority to be lowered, then the idea is: 1. at start-up create a heartbeat thread at the normal priority 2. lower the priority of the main VM thread below the heartbeat thread. Alternatively, one could spawn a new lower-priority thread to run Smalltalk code, but this may be be much more work.

The draw-back is that running Smalltalk in a thread whose priority is lower than the default *might* impact performance with lots of other processes running. This depends on whether the scheduler conflates thread priorities with process priorities (which was the default with old linux threads, which were akin to processes).

Sop there are some tests to perform:

a) see if one can lower the priority of a thread without having a /etc/security/limits.d/program.conf in place b) write a simple performance test (nfib?) in a program that can be run either with its thread having normal or lower priority, and run two instances of the program at the same time and see if they take significantly different times to compute their result

If a) is possible and b) shows no significant difference in the wall-times of the two programs then we can modify the linux heartbeat code to *lower* the priority of the main Smalltalk thread if it finds it can't create a heartbeat thread with higher priority.

I hope this answers your questions.

As a footnote let me describe why we use a heartbeat at all. When I started working on the VisualWorks VM (HPS) in the '90s it had no heartbeat (IIRC, it might have only been the Windows VM that worked like this). Instead there was a counter decremented in every frame-building send (i.e. in the jitted machine code that activated a Smalltalk send), and when this counter went to zero the VM broke out and checked for events. This counter was initialized to 256 (IIRC). Consequently there was an enormous frequency of event checks, until, that is, ione did something that reduced the frequency of frame-building sends. One day I was doing something which invoked lots of long-running large integer primitives and I noticed that when I tried to interrupt the program it took many seconds before the system stopped. What was happening was that the large integer primitives were taking so long that the counter took many seconds to count down to 0. The system didn't check for events very often. So the problems with a counter are that a) a read-modify-write cycle for a counter is in itself very expensive in a high-frequency operation like building a frame b) in normal operation the counter causes far too many check-fore-event calls c) in abnormal operation the counter causes infrequent check-fore-event calls

One solution on Unix is an interval timer (which my old BrouHaHa VMs used, but it did;t have much of an FFI so the problems it caused weren't pressing).

The natural solution is a heartbeat thread, and this is used in a number of VMs. One gets a regular event check frequency at very low cost. In Smalltalk VMs which do context-to-stack mapping it is natural to organize the stack as a set of pages and hence to have frame building sends check a stack limit (guarding the end of the page). The heartbeat simply sets the stack limit to the highest possible address to cause a stack limit check failure on the next send, and the stack check failure code checks if the stack limit has been set to the highest dress and calls the event check instead of handling the stack page overflow. In the HotSpot Java VM, if the platform supports it, a frame building send writes a byte to a guard page. Modern professors have write buffers so the write has very low cost (because it is never read) and is effectively free. So the heartbeat changes the guard page's permissions to take away write permission and cause an exception. The exception handler then checks and causes the VM to check for events. For this to work, all of writes, removing and setting page write permissions and handling exceptions must be sufficiently cheap. Anyone looking for a low-level project for the Cog VM could take a look at this mechanism. I've chosen to stick with the simple stack limit approach.

Fabio

...

To summarize, the itimer heartbeat is to be avoided as much as possible. It causes hard to debug issues with external code, has to be turned off and on around fork. It's a stop gap. Having to install a file in /etc just to be able to use a thread is insane (and AFAICT unique to linux). Whatever you do in the short term to deal with these problems I'll support, but in the long term we simply want a threaded heartbeat without needing to install anything.

...

The code in sqUnixHeartbeat.c is not a lot nor very complex, it should

not be difficult to do...

...

Also, what would be the drawbacks besides an increase on the vm size?

I hope I've explained above that I expect the drawbacks will be intermittent failures of external code.

...

Guille

_,,,^..^,,,_ best, Eliot

Hi Eliot,

On Fri, Jan 6, 2017 at 8:23 PM Eliot Miranda eliot.miranda@gmail.com wrote:

Hi Fabio, Hi Guille,

On Fri, Jan 6, 2017 at 9:44 AM, Fabio Niephaus lists@fniephaus.com wrote:

On Fri, Jan 6, 2017 at 6:33 PM Eliot Miranda eliot.miranda@gmail.com wrote:

Hi Guille,

...

On Jan 6, 2017, at 6:44 AM, Guillermo Polito guillermopolito@gmail.com

wrote:

...

Hi,

I was checking the code in sqUnixHeartbeat.c to see how the heartbeat

thread/itimer worked. It somehow bothers me that there are different compiled artifacts, one per option.

...

What do you think about having a VM that manages that as an argument

provided when we launch the VM? This would add some flexibility that we don't have right now because we make the decision at compile time.

I think it's a fine idea but it isn't really the issue. The issue is that the itimer mechanism is problematic, especially for foreign code, and is therefore a stop gap. The itimer interrupts long-running system calls, which means that things like sound libraries break (at Qwaq I had to fix ALSA to get it to work with the itimer heartbeat). Since Pharo is becoming more reliant on external code it may impact us more going forward.

The real issue is that linux's requirement that thread priorities be set in per-application file in /etc/security/limits.d (IIRC) is a big. Neither Windows nor Mac OS X requires such nonsense, and a threaded heartbeat is used on those systems without any issue at all. Why linux erected this mess in the first place is something I don't understand.

I had to implement the itimer heartbeat to get Qwaq forums running on Linux running pre 2.6 kernels, but had many other problems to solve as a result (ALSA, database connects).

Were it that the vm merely had to detect whether it could use the threaded heartbeat then things would be easy. Instead one can only use the thing if one has superuser permissions to install a file in /etc, just to use a thread of higher priority than the main one.

Thanks for the explanation, Eliot. I had no idea how bad the issues are with the itimer, but I'm glad you also see the user-facing issue with the heartbeat.

An alternative might be to lower the priority of the main thread. Then the file installation would be unnecessary.

Could you elaborate a little bit more on this idea? How could this impact the vm? What could be the drawbacks here?

First of all, for the heartbeat thread to work reliably it must run at higher priority than the thread running Smalltalk code. This is because its job is to cause Smalltalk code to break out at regular intervals to check for events. If the Smalltalk code is compute-intensive then it will prevent the heartbeat thread from running unless the heartbeat thread is running at a higher priority, and so it will be impossible to receive input keys, etc. (Note that if event collection was in a separate thread it would suffer the same issue; compute intensive code would block the event collection thread unless it was running at higher priority).

Right now, Linux restricts creating threads with priority higher than the default to those programs that have a /etc/security/limits.d/program.conf file that specifies the highest priority thread the program can create. And prior to the 2.6.12 kernel only superuser processes could create higher-priority threads. I do know that prior to 2.6.12 one couldn't create threads of *lower* priority than the default either (I would have used this if I could).

If 2.6.12 allows a program to create threads with lower priorities *without* needing a /etc/security/limits.d/program.conf, or more conveniently to allow a thread's priority to be lowered, then the idea is:

1. at start-up create a heartbeat thread at the normal priority
2. lower the priority of the main VM thread below the heartbeat thread.

Alternatively, one could spawn a new lower-priority thread to run Smalltalk code, but this may be be much more work.

The draw-back is that running Smalltalk in a thread whose priority is lower than the default *might* impact performance with lots of other processes running. This depends on whether the scheduler conflates thread priorities with process priorities (which was the default with old linux threads, which were akin to processes).

Sop there are some tests to perform:

a) see if one can lower the priority of a thread without having a /etc/security/limits.d/program.conf in place b) write a simple performance test (nfib?) in a program that can be run either with its thread having normal or lower priority, and run two instances of the program at the same time and see if they take significantly different times to compute their result

If a) is possible and b) shows no significant difference in the wall-times of the two programs then we can modify the linux heartbeat code to *lower* the priority of the main Smalltalk thread if it finds it can't create a heartbeat thread with higher priority.

I hope this answers your questions.

Yes, it does. Thanks!

As a footnote let me describe why we use a heartbeat at all. When I started working on the VisualWorks VM (HPS) in the '90s it had no heartbeat (IIRC, it might have only been the Windows VM that worked like this). Instead there was a counter decremented in every frame-building send (i.e. in the jitted machine code that activated a Smalltalk send), and when this counter went to zero the VM broke out and checked for events. This counter was initialized to 256 (IIRC). Consequently there was an enormous frequency of event checks, until, that is, ione did something that reduced the frequency of frame-building sends. One day I was doing something which invoked lots of long-running large integer primitives and I noticed that when I tried to interrupt the program it took many seconds before the system stopped. What was happening was that the large integer primitives were taking so long that the counter took many seconds to count down to 0. The system didn't check for events very often. So the problems with a counter are that a) a read-modify-write cycle for a counter is in itself very expensive in a high-frequency operation like building a frame b) in normal operation the counter causes far too many check-fore-event calls c) in abnormal operation the counter causes infrequent check-fore-event calls

One solution on Unix is an interval timer (which my old BrouHaHa VMs used, but it did;t have much of an FFI so the problems it caused weren't pressing).

The natural solution is a heartbeat thread, and this is used in a number of VMs. One gets a regular event check frequency at very low cost. In Smalltalk VMs which do context-to-stack mapping it is natural to organize the stack as a set of pages and hence to have frame building sends check a stack limit (guarding the end of the page). The heartbeat simply sets the stack limit to the highest possible address to cause a stack limit check failure on the next send, and the stack check failure code checks if the stack limit has been set to the highest dress and calls the event check instead of handling the stack page overflow. In the HotSpot Java VM, if the platform supports it, a frame building send writes a byte to a guard page. Modern professors have write buffers so the write has very low cost (because it is never read) and is effectively free. So the heartbeat changes the guard page's permissions to take away write permission and cause an exception. The exception handler then checks and causes the VM to check for events. For this to work, all of writes, removing and setting page write permissions and handling exceptions must be sufficiently cheap. Anyone looking for a low-level project for the Cog VM could take a look at this mechanism. I've chosen to stick with the simple stack limit approach.

And thanks for all this info. I didn't really know much about the itimer vs heartbeat thread topic TBH. I was just surprised that this is so complicated, because I thought that event handling would be relatively easy. However, I still don't completely understand why other applications (e.g. games) don't have these event problems, but that's something I can look into myself this weekend :)

Fabio

Fabio

To summarize, the itimer heartbeat is to be avoided as much as possible. It causes hard to debug issues with external code, has to be turned off and on around fork. It's a stop gap. Having to install a file in /etc just to be able to use a thread is insane (and AFAICT unique to linux). Whatever you do in the short term to deal with these problems I'll support, but in the long term we simply want a threaded heartbeat without needing to install anything.

...

The code in sqUnixHeartbeat.c is not a lot nor very complex, it should

not be difficult to do...

...

Also, what would be the drawbacks besides an increase on the vm size?

I hope I've explained above that I expect the drawbacks will be intermittent failures of external code.

...

Guille

_,,,^..^,,,_ best, Eliot

On Tue, Mar 14, 2017 at 7:48 AM Eliot Miranda eliot.miranda@gmail.com wrote:

Hi Fabio,

On Jan 6, 2017, at 1:47 PM, Fabio Niephaus lists@fniephaus.com wrote:

Hi Eliot,

On Fri, Jan 6, 2017 at 8:23 PM Eliot Miranda eliot.miranda@gmail.com wrote:

Hi Fabio, Hi Guille,

On Fri, Jan 6, 2017 at 9:44 AM, Fabio Niephaus lists@fniephaus.com wrote:

On Fri, Jan 6, 2017 at 6:33 PM Eliot Miranda eliot.miranda@gmail.com wrote:

Hi Guille,

...

On Jan 6, 2017, at 6:44 AM, Guillermo Polito guillermopolito@gmail.com

wrote:

...

Hi,

I was checking the code in sqUnixHeartbeat.c to see how the heartbeat

thread/itimer worked. It somehow bothers me that there are different compiled artifacts, one per option.

...

What do you think about having a VM that manages that as an argument

provided when we launch the VM? This would add some flexibility that we don't have right now because we make the decision at compile time.

I think it's a fine idea but it isn't really the issue. The issue is that the itimer mechanism is problematic, especially for foreign code, and is therefore a stop gap. The itimer interrupts long-running system calls, which means that things like sound libraries break (at Qwaq I had to fix ALSA to get it to work with the itimer heartbeat). Since Pharo is becoming more reliant on external code it may impact us more going forward.

The real issue is that linux's requirement that thread priorities be set in per-application file in /etc/security/limits.d (IIRC) is a big. Neither Windows nor Mac OS X requires such nonsense, and a threaded heartbeat is used on those systems without any issue at all. Why linux erected this mess in the first place is something I don't understand.

I had to implement the itimer heartbeat to get Qwaq forums running on Linux running pre 2.6 kernels, but had many other problems to solve as a result (ALSA, database connects).

Were it that the vm merely had to detect whether it could use the threaded heartbeat then things would be easy. Instead one can only use the thing if one has superuser permissions to install a file in /etc, just to use a thread of higher priority than the main one.

Thanks for the explanation, Eliot. I had no idea how bad the issues are with the itimer, but I'm glad you also see the user-facing issue with the heartbeat.

An alternative might be to lower the priority of the main thread. Then the file installation would be unnecessary.

Could you elaborate a little bit more on this idea? How could this impact the vm? What could be the drawbacks here?

First of all, for the heartbeat thread to work reliably it must run at higher priority than the thread running Smalltalk code. This is because its job is to cause Smalltalk code to break out at regular intervals to check for events. If the Smalltalk code is compute-intensive then it will prevent the heartbeat thread from running unless the heartbeat thread is running at a higher priority, and so it will be impossible to receive input keys, etc. (Note that if event collection was in a separate thread it would suffer the same issue; compute intensive code would block the event collection thread unless it was running at higher priority).

Right now, Linux restricts creating threads with priority higher than the default to those programs that have a /etc/security/limits.d/program.conf file that specifies the highest priority thread the program can create. And prior to the 2.6.12 kernel only superuser processes could create higher-priority threads. I do know that prior to 2.6.12 one couldn't create threads of *lower* priority than the default either (I would have used this if I could).

If 2.6.12 allows a program to create threads with lower priorities *without* needing a /etc/security/limits.d/program.conf, or more conveniently to allow a thread's priority to be lowered, then the idea is:

1. at start-up create a heartbeat thread at the normal priority
2. lower the priority of the main VM thread below the heartbeat thread.

Alternatively, one could spawn a new lower-priority thread to run Smalltalk code, but this may be be much more work.

The draw-back is that running Smalltalk in a thread whose priority is lower than the default *might* impact performance with lots of other processes running. This depends on whether the scheduler conflates thread priorities with process priorities (which was the default with old linux threads, which were akin to processes).

Sop there are some tests to perform:

a) see if one can lower the priority of a thread without having a /etc/security/limits.d/program.conf in place b) write a simple performance test (nfib?) in a program that can be run either with its thread having normal or lower priority, and run two instances of the program at the same time and see if they take significantly different times to compute their result

If a) is possible and b) shows no significant difference in the wall-times of the two programs then we can modify the linux heartbeat code to *lower* the priority of the main Smalltalk thread if it finds it can't create a heartbeat thread with higher priority.

I hope this answers your questions.

Yes, it does. Thanks!

As a footnote let me describe why we use a heartbeat at all. When I started working on the VisualWorks VM (HPS) in the '90s it had no heartbeat (IIRC, it might have only been the Windows VM that worked like this). Instead there was a counter decremented in every frame-building send (i.e. in the jitted machine code that activated a Smalltalk send), and when this counter went to zero the VM broke out and checked for events. This counter was initialized to 256 (IIRC). Consequently there was an enormous frequency of event checks, until, that is, ione did something that reduced the frequency of frame-building sends. One day I was doing something which invoked lots of long-running large integer primitives and I noticed that when I tried to interrupt the program it took many seconds before the system stopped. What was happening was that the large integer primitives were taking so long that the counter took many seconds to count down to 0. The system didn't check for events very often. So the problems with a counter are that a) a read-modify-write cycle for a counter is in itself very expensive in a high-frequency operation like building a frame b) in normal operation the counter causes far too many check-fore-event calls c) in abnormal operation the counter causes infrequent check-fore-event calls

One solution on Unix is an interval timer (which my old BrouHaHa VMs used, but it did;t have much of an FFI so the problems it caused weren't pressing).

The natural solution is a heartbeat thread, and this is used in a number of VMs. One gets a regular event check frequency at very low cost. In Smalltalk VMs which do context-to-stack mapping it is natural to organize the stack as a set of pages and hence to have frame building sends check a stack limit (guarding the end of the page). The heartbeat simply sets the stack limit to the highest possible address to cause a stack limit check failure on the next send, and the stack check failure code checks if the stack limit has been set to the highest dress and calls the event check instead of handling the stack page overflow. In the HotSpot Java VM, if the platform supports it, a frame building send writes a byte to a guard page. Modern professors have write buffers so the write has very low cost (because it is never read) and is effectively free. So the heartbeat changes the guard page's permissions to take away write permission and cause an exception. The exception handler then checks and causes the VM to check for events. For this to work, all of writes, removing and setting page write permissions and handling exceptions must be sufficiently cheap. Anyone looking for a low-level project for the Cog VM could take a look at this mechanism. I've chosen to stick with the simple stack limit approach.

And thanks for all this info. I didn't really know much about the itimer vs heartbeat thread topic TBH. I was just surprised that this is so complicated, because I thought that event handling would be relatively easy. However, I still don't completely understand why other applications (e.g. games) don't have these event problems, but that's something I can look into myself this weekend :)

Forgive me; I missed this excellent question. I don't know if I'm correct but I presume the the reason a game application has no difficulties like this is that it can include an explicit poll for events as part of its main loop. While a game may be quite complex with many moving parts it's overall architecture is quite simple. It is a loop, advancing a model, rendering the model, and polling for events.

A Smalltalk vm, like a real processor, is a compute engineer me that goes whatever it is told, which may include being told to do nothing but execute code, but it must still be able to receive interrupts, even when running at full tilt performing nothing but calculations.

If one wants the Smalltalk vm to run suboptimally one will arrange that as part of its normal execution (e.g. on method activations) it will burn some cycles deciding if it should poll for events, e.g. decrenentung a counter. But as I explained elsewhere this has problems. If method invocation frequency falls, e.g. as a side effect of invoking long-running primitives, then it may poll for events too infrequently.

If however, one wants the Smalltalk vm to run optimally, one wants the event check to occur at regular intervals, without slowly Ng down Smalltalk execution, hence either the itimer interrupt or the heartbeat thread.

Does this make things clearer?

Hi Eliot, Yes, it makes sense when you keep in mind that the VM aims to behave like a processor. Thanks a lot!

Best, Fabio

Fabio

Eliot _,,,^..^,,,_ (phone)

Fabio

To summarize, the itimer heartbeat is to be avoided as much as possible. It causes hard to debug issues with external code, has to be turned off and on around fork. It's a stop gap. Having to install a file in /etc just to be able to use a thread is insane (and AFAICT unique to linux). Whatever you do in the short term to deal with these problems I'll support, but in the long term we simply want a threaded heartbeat without needing to install anything.

...

The code in sqUnixHeartbeat.c is not a lot nor very complex, it should

not be difficult to do...

...

Also, what would be the drawbacks besides an increase on the vm size?

I hope I've explained above that I expect the drawbacks will be intermittent failures of external code.

...

Guille

_,,,^..^,,,_ best, Eliot

Hi Denis,

On Tue, Jan 10, 2017 at 12:08 PM, Denis Kudriashov dionisiydk@gmail.com wrote:

Hi.

Will event-driven VM fix this problem completely? Or heartbeat will be needed anyway?

As I understand it, the heartbeat us always needed. In an event-driven VM it may be that the Smalltalk executive is called form the event loop, but the Smalltalk executive still has to break out of executing Smalltalk to return to the event loop to receive new events.

The thing to understand about the JIT VM is that Smalltalk is executing in machine code just like simple code in some low level language. That code can respond to Smalltalk events such as attempting to wait on a Semaphore with no outstanding signals, which may cause it to switch processes. But it cannot respond to external asynchronous events unless it is informed of those events. And it is very difficult to construct a VM that can accept interrupts at arbitrary times that activate Smalltalk code (imagine receiving an interrupt in the middle of a GC, or mid-way through looking up a send not found in the cache, etc, etc; essentially interrupts can only be accepted at limited times). So the VM needs to check for events (and indeed interrupts) at safe points. The efficient implementation of safe points is checking on the next frame-building send. But if every frame-building send checked for interrupts and/or events, frame build would be very slow and the entire VM would crawl. The function of the heart beat is to cause frame building sends to check for interrupts and/or events at regular, but (relative to frame-builling send frequency) infrequent occasions.

Compare that to interrupts in a real processor. The processor /also/ only tests e.g. the interrupt request pin at a safe point (perhaps at the end of each interaction decode cycle), and also provides means to disable interrupts for critical sections of code. It's just that the quanta are much smaller than in a Smalltalk vm.

HTH

2017-01-06 20:23 GMT+01:00 Eliot Miranda eliot.miranda@gmail.com:

...

Hi Fabio, Hi Guille,

On Fri, Jan 6, 2017 at 9:44 AM, Fabio Niephaus lists@fniephaus.com wrote:

...

On Fri, Jan 6, 2017 at 6:33 PM Eliot Miranda eliot.miranda@gmail.com wrote:

...

Hi Guille,

...

On Jan 6, 2017, at 6:44 AM, Guillermo Polito <

guillermopolito@gmail.com> wrote:

...

Hi,

I was checking the code in sqUnixHeartbeat.c to see how the heartbeat

thread/itimer worked. It somehow bothers me that there are different compiled artifacts, one per option.

...

What do you think about having a VM that manages that as an argument

provided when we launch the VM? This would add some flexibility that we don't have right now because we make the decision at compile time.

I think it's a fine idea but it isn't really the issue. The issue is that the itimer mechanism is problematic, especially for foreign code, and is therefore a stop gap. The itimer interrupts long-running system calls, which means that things like sound libraries break (at Qwaq I had to fix ALSA to get it to work with the itimer heartbeat). Since Pharo is becoming more reliant on external code it may impact us more going forward.

The real issue is that linux's requirement that thread priorities be set in per-application file in /etc/security/limits.d (IIRC) is a big. Neither Windows nor Mac OS X requires such nonsense, and a threaded heartbeat is used on those systems without any issue at all. Why linux erected this mess in the first place is something I don't understand.

I had to implement the itimer heartbeat to get Qwaq forums running on Linux running pre 2.6 kernels, but had many other problems to solve as a result (ALSA, database connects).

Were it that the vm merely had to detect whether it could use the threaded heartbeat then things would be easy. Instead one can only use the thing if one has superuser permissions to install a file in /etc, just to use a thread of higher priority than the main one.

Thanks for the explanation, Eliot. I had no idea how bad the issues are with the itimer, but I'm glad you also see the user-facing issue with the heartbeat.

...

An alternative might be to lower the priority of the main thread. Then the file installation would be unnecessary.

Could you elaborate a little bit more on this idea? How could this impact the vm? What could be the drawbacks here?

First of all, for the heartbeat thread to work reliably it must run at higher priority than the thread running Smalltalk code. This is because its job is to cause Smalltalk code to break out at regular intervals to check for events. If the Smalltalk code is compute-intensive then it will prevent the heartbeat thread from running unless the heartbeat thread is running at a higher priority, and so it will be impossible to receive input keys, etc. (Note that if event collection was in a separate thread it would suffer the same issue; compute intensive code would block the event collection thread unless it was running at higher priority).

Right now, Linux restricts creating threads with priority higher than the default to those programs that have a /etc/security/limits.d/program.conf file that specifies the highest priority thread the program can create. And prior to the 2.6.12 kernel only superuser processes could create higher-priority threads. I do know that prior to 2.6.12 one couldn't create threads of *lower* priority than the default either (I would have used this if I could).

If 2.6.12 allows a program to create threads with lower priorities *without* needing a /etc/security/limits.d/program.conf, or more conveniently to allow a thread's priority to be lowered, then the idea is:

1. at start-up create a heartbeat thread at the normal priority
2. lower the priority of the main VM thread below the heartbeat thread.

Alternatively, one could spawn a new lower-priority thread to run Smalltalk code, but this may be be much more work.

The draw-back is that running Smalltalk in a thread whose priority is lower than the default *might* impact performance with lots of other processes running. This depends on whether the scheduler conflates thread priorities with process priorities (which was the default with old linux threads, which were akin to processes).

Sop there are some tests to perform:

a) see if one can lower the priority of a thread without having a /etc/security/limits.d/program.conf in place b) write a simple performance test (nfib?) in a program that can be run either with its thread having normal or lower priority, and run two instances of the program at the same time and see if they take significantly different times to compute their result

If a) is possible and b) shows no significant difference in the wall-times of the two programs then we can modify the linux heartbeat code to *lower* the priority of the main Smalltalk thread if it finds it can't create a heartbeat thread with higher priority.

I hope this answers your questions.

As a footnote let me describe why we use a heartbeat at all. When I started working on the VisualWorks VM (HPS) in the '90s it had no heartbeat (IIRC, it might have only been the Windows VM that worked like this). Instead there was a counter decremented in every frame-building send (i.e. in the jitted machine code that activated a Smalltalk send), and when this counter went to zero the VM broke out and checked for events. This counter was initialized to 256 (IIRC). Consequently there was an enormous frequency of event checks, until, that is, ione did something that reduced the frequency of frame-building sends. One day I was doing something which invoked lots of long-running large integer primitives and I noticed that when I tried to interrupt the program it took many seconds before the system stopped. What was happening was that the large integer primitives were taking so long that the counter took many seconds to count down to 0. The system didn't check for events very often. So the problems with a counter are that a) a read-modify-write cycle for a counter is in itself very expensive in a high-frequency operation like building a frame b) in normal operation the counter causes far too many check-fore-event calls c) in abnormal operation the counter causes infrequent check-fore-event calls

One solution on Unix is an interval timer (which my old BrouHaHa VMs used, but it did;t have much of an FFI so the problems it caused weren't pressing).

The natural solution is a heartbeat thread, and this is used in a number of VMs. One gets a regular event check frequency at very low cost. In Smalltalk VMs which do context-to-stack mapping it is natural to organize the stack as a set of pages and hence to have frame building sends check a stack limit (guarding the end of the page). The heartbeat simply sets the stack limit to the highest possible address to cause a stack limit check failure on the next send, and the stack check failure code checks if the stack limit has been set to the highest dress and calls the event check instead of handling the stack page overflow. In the HotSpot Java VM, if the platform supports it, a frame building send writes a byte to a guard page. Modern professors have write buffers so the write has very low cost (because it is never read) and is effectively free. So the heartbeat changes the guard page's permissions to take away write permission and cause an exception. The exception handler then checks and causes the VM to check for events. For this to work, all of writes, removing and setting page write permissions and handling exceptions must be sufficiently cheap. Anyone looking for a low-level project for the Cog VM could take a look at this mechanism. I've chosen to stick with the simple stack limit approach.

Fabio

...

...

To summarize, the itimer heartbeat is to be avoided as much as possible. It causes hard to debug issues with external code, has to be turned off and on around fork. It's a stop gap. Having to install a file in /etc just to be able to use a thread is insane (and AFAICT unique to linux). Whatever you do in the short term to deal with these problems I'll support, but in the long term we simply want a threaded heartbeat without needing to install anything.

...

The code in sqUnixHeartbeat.c is not a lot nor very complex, it

should not be difficult to do...

...

Also, what would be the drawbacks besides an increase on the vm size?

I hope I've explained above that I expect the drawbacks will be intermittent failures of external code.

...

Guille

_,,,^..^,,,_ best, Eliot

On Wed, Jan 11, 2017 at 4:54 AM, Eliot Miranda eliot.miranda@gmail.com wrote:

Hi Denis,

On Tue, Jan 10, 2017 at 12:08 PM, Denis Kudriashov dionisiydk@gmail.com wrote:

...

Hi.

Will event-driven VM fix this problem completely? Or heartbeat will be needed anyway?

As I understand it, the heartbeat us always needed. In an event-driven VM it may be that the Smalltalk executive is called form the event loop, but the Smalltalk executive still has to break out of executing Smalltalk to return to the event loop to receive new events.

The thing to understand about the JIT VM is that Smalltalk is executing in machine code just like simple code in some low level language. That code can respond to Smalltalk events such as attempting to wait on a Semaphore with no outstanding signals, which may cause it to switch processes. But it cannot respond to external asynchronous events unless it is informed of those events. And it is very difficult to construct a VM that can accept interrupts at arbitrary times that activate Smalltalk code (imagine receiving an interrupt in the middle of a GC, or mid-way through looking up a send not found in the cache, etc, etc; essentially interrupts can only be accepted at limited times). So the VM needs to check for events (and indeed interrupts) at safe points. The efficient implementation of safe points is checking on the next frame-building send. But if every frame-building send checked for interrupts and/or events, frame build would be very slow and the entire VM would crawl. The function of the heart beat is to cause frame building sends to check for interrupts and/or events at regular, but (relative to frame-builling send frequency) infrequent occasions.

Compare that to interrupts in a real processor. The processor /also/ only tests e.g. the interrupt request pin at a safe point (perhaps at the end of each interaction decode cycle), and also provides means to disable interrupts for critical sections of code. It's just that the quanta are much smaller than in a Smalltalk vm.

HTH

2017-01-06 20:23 GMT+01:00 Eliot Miranda eliot.miranda@gmail.com:

...

...

Hi Fabio, Hi Guille,

On Fri, Jan 6, 2017 at 9:44 AM, Fabio Niephaus lists@fniephaus.com wrote:

...

On Fri, Jan 6, 2017 at 6:33 PM Eliot Miranda eliot.miranda@gmail.com wrote:

...

Hi Guille,

...

On Jan 6, 2017, at 6:44 AM, Guillermo Polito <

guillermopolito@gmail.com> wrote:

...

Hi,

I was checking the code in sqUnixHeartbeat.c to see how the

heartbeat thread/itimer worked. It somehow bothers me that there are different compiled artifacts, one per option.

...

What do you think about having a VM that manages that as an argument

provided when we launch the VM? This would add some flexibility that we don't have right now because we make the decision at compile time.

I think it's a fine idea but it isn't really the issue. The issue is that the itimer mechanism is problematic, especially for foreign code, and is therefore a stop gap. The itimer interrupts long-running system calls, which means that things like sound libraries break (at Qwaq I had to fix ALSA to get it to work with the itimer heartbeat). Since Pharo is becoming more reliant on external code it may impact us more going forward.

The real issue is that linux's requirement that thread priorities be set in per-application file in /etc/security/limits.d (IIRC) is a big. Neither Windows nor Mac OS X requires such nonsense, and a threaded heartbeat is used on those systems without any issue at all. Why linux erected this mess in the first place is something I don't understand.

I had to implement the itimer heartbeat to get Qwaq forums running on Linux running pre 2.6 kernels, but had many other problems to solve as a result (ALSA, database connects).

Were it that the vm merely had to detect whether it could use the threaded heartbeat then things would be easy. Instead one can only use the thing if one has superuser permissions to install a file in /etc, just to use a thread of higher priority than the main one.

Thanks for the explanation, Eliot. I had no idea how bad the issues are with the itimer, but I'm glad you also see the user-facing issue with the heartbeat.

...

An alternative might be to lower the priority of the main thread. Then the file installation would be unnecessary.

Could you elaborate a little bit more on this idea? How could this impact the vm? What could be the drawbacks here?

First of all, for the heartbeat thread to work reliably it must run at higher priority than the thread running Smalltalk code. This is because its job is to cause Smalltalk code to break out at regular intervals to check for events. If the Smalltalk code is compute-intensive then it will prevent the heartbeat thread from running unless the heartbeat thread is running at a higher priority, and so it will be impossible to receive input keys, etc. (Note that if event collection was in a separate thread it would suffer the same issue; compute intensive code would block the event collection thread unless it was running at higher priority).

Right now, Linux restricts creating threads with priority higher than the default to those programs that have a /etc/security/limits.d/program.conf file that specifies the highest priority thread the program can create. And prior to the 2.6.12 kernel only superuser processes could create higher-priority threads. I do know that prior to 2.6.12 one couldn't create threads of *lower* priority than the default either (I would have used this if I could).

If 2.6.12 allows a program to create threads with lower priorities *without* needing a /etc/security/limits.d/program.conf, or more conveniently to allow a thread's priority to be lowered, then the idea is:

1. at start-up create a heartbeat thread at the normal priority
2. lower the priority of the main VM thread below the heartbeat thread.

Alternatively, one could spawn a new lower-priority thread to run Smalltalk code, but this may be be much more work.

The draw-back is that running Smalltalk in a thread whose priority is lower than the default *might* impact performance with lots of other processes running. This depends on whether the scheduler conflates thread priorities with process priorities (which was the default with old linux threads, which were akin to processes).

Sop there are some tests to perform:

a) see if one can lower the priority of a thread without having a /etc/security/limits.d/program.conf in place b) write a simple performance test (nfib?) in a program that can be run either with its thread having normal or lower priority, and run two instances of the program at the same time and see if they take significantly different times to compute their result

If a) is possible and b) shows no significant difference in the wall-times of the two programs then we can modify the linux heartbeat code to *lower* the priority of the main Smalltalk thread if it finds it can't create a heartbeat thread with higher priority.

With recent Pharo discussion on heatbeat itimer versus threaded

http://forum.world.st/Esteban-s-ChangeLog-week-of-6-March-2017-td4938349.htm...

I got curious....

$ vi test.c #include <stdio.h> #include <sys/time.h> #include <sys/resource.h> #include <stdlib.h> #include <time.h>

void fib(int n, int pid) { int first = 0, second = 1, next, c; for ( c = 0 ; c < n ; c++ ) { if ( c <= 1 ) next = c; else { next = first + second; first = second; second = next; } //printf("%d - fib=%d\n", pid, next); } }

struct timespec timediff(struct timespec start, struct timespec end) { struct timespec temp; temp.tv_sec = end.tv_sec-start.tv_sec; temp.tv_nsec = end.tv_nsec-start.tv_nsec; if (temp.tv_nsec < 0) { temp.tv_nsec += 1000000000; temp.tv_sec -= 1; } return temp; }

int main(int argc, char *argv[]) { int which = PRIO_PROCESS; id_t pid; int priority, fibN, ret; struct timespec start, stop, diff;

if(argc<2) { printf("Usage: %s ProcessPriority FibN\n", argv[0]); exit(1); } priority = atoi(argv[1]); fibN = atoi(argv[2]);

pid = getpid(); printf("%d\t ==> %d original priority\n", pid, getpriority(which, pid));

setpriority(which, pid, priority); priority = getpriority(which, pid); printf("%d\t ==> %d new priority\n", pid, priority);

clock_gettime( CLOCK_REALTIME, &start); sleep(1); // allow all threads to be scheduled fib(fibN, pid); clock_gettime( CLOCK_REALTIME, &stop); diff = timediff(start, stop); printf("\n%d @ %d\t ==> execution time %d:%d\n", pid, priority, diff.tv_sec - 1, diff.tv_nsec); }

///////////////////////

$ gcc test.c

$ uname -a Linux dom0 3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) i686 GNU/Linux

$ nproc --all 4

$ N=1000000000 ; for NPROC in 1 ; do (./a.out 19 $N &) && (./a.out 1 $N &) && (./a.out 0 $N &) ; done 28175 @ 0 ==> execution time 5:137392274 28171 @ 19 ==> execution time 5:493271222 28173 @ 1 ==> execution time 5:678498982

...for NPROC in 1 2 ; do... 28339 @ 0 ==> execution time 5:891516242 28333 @ 0 ==> execution time 6:101871486 28331 @ 1 ==> execution time 6:197583303 28337 @ 1 ==> execution time 6:473926938 28335 @ 19 ==> execution time 11:19093473 28329 @ 19 ==> execution time 11:109494611

...for NPROC in 1 2 3 ; do... 28370 @ 0 ==> execution time 8:286661748 28364 @ 0 ==> execution time 8:346971535 28376 @ 0 ==> execution time 8:919760746 28362 @ 1 ==> execution time 9:943310436 28368 @ 1 ==> execution time 10:43977329 28374 @ 1 ==> execution time 10:251189507 28372 @ 19 ==> execution time 14:807238482 28360 @ 19 ==> execution time 15:48684466 28366 @ 19 ==> execution time 15:392610447

...for NPROC in 1 2 3 4 ; do... 28401 @ 0 ==> execution time 10:808863373 28407 @ 0 ==> execution time 11:144571568 28395 @ 0 ==> execution time 11:311897577 28389 @ 0 ==> execution time 12:49899167 28399 @ 1 ==> execution time 12:391939682 28387 @ 1 ==> execution time 12:922309497 28393 @ 1 ==> execution time 12:997908723 28405 @ 1 ==> execution time 13:116623935 28385 @ 19 ==> execution time 18:710195627 28391 @ 19 ==> execution time 19:160867082 28397 @ 19 ==> execution time 19:306339215 28403 @ 19 ==> execution time 19:340283641

...for NPROC in 1 2 3 4 5; do... 28431 @ 0 ==> execution time 13:512767819 28437 @ 0 ==> execution time 13:682814129 28449 @ 0 ==> execution time 13:695500843 28443 @ 0 ==> execution time 14:212788768 28425 @ 0 ==> execution time 14:384973010 28435 @ 1 ==> execution time 15:356897222 28423 @ 1 ==> execution time 15:638388578 28441 @ 1 ==> execution time 15:692913828 28429 @ 1 ==> execution time 15:822047741 28447 @ 1 ==> execution time 15:845915440 28445 @ 19 ==> execution time 23:221173838 28433 @ 19 ==> execution time 23:349862424 28421 @ 19 ==> execution time 23:388930822 28439 @ 19 ==> execution time 23:374234661 28427 @ 19 ==> execution time 23:517771887

cheers -ben

Hi Fabio, Hi Guille,

On Fri, Jan 6, 2017 at 9:44 AM, Fabio Niephaus lists@fniephaus.com wrote:

...

On Fri, Jan 6, 2017 at 6:33 PM Eliot Miranda eliot.miranda@gmail.com wrote:

...

Hi Guille,

...

On Jan 6, 2017, at 6:44 AM, Guillermo Polito guillermopolito@gmail.com

wrote:

...

Hi,

I was checking the code in sqUnixHeartbeat.c to see how the heartbeat

thread/itimer worked. It somehow bothers me that there are different compiled artifacts, one per option.

...

What do you think about having a VM that manages that as an argument

provided when we launch the VM? This would add some flexibility that we don't have right now because we make the decision at compile time.

I think it's a fine idea but it isn't really the issue. The issue is that the itimer mechanism is problematic, especially for foreign code, and is therefore a stop gap. The itimer interrupts long-running system calls, which means that things like sound libraries break (at Qwaq I had to fix ALSA to get it to work with the itimer heartbeat). Since Pharo is becoming more reliant on external code it may impact us more going forward.

The real issue is that linux's requirement that thread priorities be set in per-application file in /etc/security/limits.d (IIRC) is a big. Neither Windows nor Mac OS X requires such nonsense, and a threaded heartbeat is used on those systems without any issue at all. Why linux erected this mess in the first place is something I don't understand.

I had to implement the itimer heartbeat to get Qwaq forums running on Linux running pre 2.6 kernels, but had many other problems to solve as a result (ALSA, database connects).

Were it that the vm merely had to detect whether it could use the threaded heartbeat then things would be easy. Instead one can only use the thing if one has superuser permissions to install a file in /etc, just to use a thread of higher priority than the main one.

Thanks for the explanation, Eliot. I had no idea how bad the issues are with the itimer, but I'm glad you also see the user-facing issue with the heartbeat.

...

An alternative might be to lower the priority of the main thread. Then the file installation would be unnecessary.

Could you elaborate a little bit more on this idea? How could this impact the vm? What could be the drawbacks here?

First of all, for the heartbeat thread to work reliably it must run at higher priority than the thread running Smalltalk code. This is because its job is to cause Smalltalk code to break out at regular intervals to check for events. If the Smalltalk code is compute-intensive then it will prevent the heartbeat thread from running unless the heartbeat thread is running at a higher priority, and so it will be impossible to receive input keys, etc. (Note that if event collection was in a separate thread it would suffer the same issue; compute intensive code would block the event collection thread unless it was running at higher priority).

Right now, Linux restricts creating threads with priority higher than the default to those programs that have a /etc/security/limits.d/program.conf file that specifies the highest priority thread the program can create. And prior to the 2.6.12 kernel only superuser processes could create higher-priority threads. I do know that prior to 2.6.12 one couldn't create threads of *lower* priority than the default either (I would have used this if I could).

If 2.6.12 allows a program to create threads with lower priorities *without* needing a /etc/security/limits.d/program.conf, or more conveniently to allow a thread's priority to be lowered, then the idea is:

1. at start-up create a heartbeat thread at the normal priority
2. lower the priority of the main VM thread below the heartbeat thread.

Alternatively, one could spawn a new lower-priority thread to run Smalltalk code, but this may be be much more work.

The draw-back is that running Smalltalk in a thread whose priority is lower than the default *might* impact performance with lots of other processes running. This depends on whether the scheduler conflates thread priorities with process priorities (which was the default with old linux threads, which were akin to processes).

If VM will have option for --heartbeat vs --itimer, can be this heartbeat thread priorities as option too?

There is not only problems with older Linux kernels, there is also problems with Docker (some complaints in this thread) and also FreeBSD - VM needs to be run as root (due to higher thread priorities), which is unusable in real world (security).

Sop there are some tests to perform:

a) see if one can lower the priority of a thread without having a /etc/security/limits.d/program.conf in place b) write a simple performance test (nfib?) in a program that can be run either with its thread having normal or lower priority, and run two instances of the program at the same time and see if they take significantly different times to compute their result

If a) is possible and b) shows no significant difference in the wall-times of the two programs then we can modify the linux heartbeat code to *lower* the priority of the main Smalltalk thread if it finds it can't create a heartbeat thread with higher priority.

I hope this answers your questions.

As a footnote let me describe why we use a heartbeat at all. When I started working on the VisualWorks VM (HPS) in the '90s it had no heartbeat (IIRC, it might have only been the Windows VM that worked like this). Instead there was a counter decremented in every frame-building send (i.e. in the jitted machine code that activated a Smalltalk send), and when this counter went to zero the VM broke out and checked for events. This counter was initialized to 256 (IIRC). Consequently there was an enormous frequency of event checks, until, that is, ione did something that reduced the frequency of frame-building sends. One day I was doing something which invoked lots of long-running large integer primitives and I noticed that when I tried to interrupt the program it took many seconds before the system stopped. What was happening was that the large integer primitives were taking so long that the counter took many seconds to count down to 0. The system didn't check for events very often. So the problems with a counter are that a) a read-modify-write cycle for a counter is in itself very expensive in a high-frequency operation like building a frame b) in normal operation the counter causes far too many check-fore-event calls c) in abnormal operation the counter causes infrequent check-fore-event calls

One solution on Unix is an interval timer (which my old BrouHaHa VMs used, but it did;t have much of an FFI so the problems it caused weren't pressing).

The natural solution is a heartbeat thread, and this is used in a number of VMs. One gets a regular event check frequency at very low cost. In Smalltalk VMs which do context-to-stack mapping it is natural to organize the stack as a set of pages and hence to have frame building sends check a stack limit (guarding the end of the page). The heartbeat simply sets the stack limit to the highest possible address to cause a stack limit check failure on the next send, and the stack check failure code checks if the stack limit has been set to the highest dress and calls the event check instead of handling the stack page overflow. In the HotSpot Java VM, if the platform supports it, a frame building send writes a byte to a guard page. Modern professors have write buffers so the write has very low cost (because it is never read) and is effectively free. So the heartbeat changes the guard page's permissions to take away write permission and cause an exception. The exception handler then checks and causes the VM to check for events. For this to work, all of writes, removing and setting page write permissions and handling exceptions must be sufficiently cheap. Anyone looking for a low-level project for the Cog VM could take a look at this mechanism. I've chosen to stick with the simple stack limit approach.

Fabio

...

...

To summarize, the itimer heartbeat is to be avoided as much as possible. It causes hard to debug issues with external code, has to be turned off and on around fork. It's a stop gap. Having to install a file in /etc just to be able to use a thread is insane (and AFAICT unique to linux). Whatever you do in the short term to deal with these problems I'll support, but in the long term we simply want a threaded heartbeat without needing to install anything.

...

The code in sqUnixHeartbeat.c is not a lot nor very complex, it should

not be difficult to do...

...

Also, what would be the drawbacks besides an increase on the vm size?

I hope I've explained above that I expect the drawbacks will be intermittent failures of external code.

...

Guille

_,,,^..^,,,_ best, Eliot

Hi Ben,

On Mon, Mar 20, 2017 at 9:43 AM, Ben Coman btc@openinworld.com wrote:

On Sat, Jan 7, 2017 at 3:23 AM, Eliot Miranda eliot.miranda@gmail.com wrote:

...

First of all, for the heartbeat thread to work reliably it must run at higher priority than the thread running Smalltalk code.

After reading pages 11-14 of [1] , page 11 of [2] , and then [3] I was wondering if the dynamic priorities of the Linux 2.6 scheduler would cause a sleeping heartbeat thread to *effectively* remain a higher priority than the Smalltalk execution thread.

[1] http://www.isical.ac.in/~mandar/os/scheduling-linux.pdf [2] https://www.cs.columbia.edu/~smb/classes/s06-4118/l13.pdf [3] http://stackoverflow.com/questions/25577403/how-are- dynamic-priorities-of-threads-computed-in-linux-2-6-x

...

This is because its job is to cause Smalltalk code to break out at regular intervals to check for events. If the Smalltalk code is compute-intensive then it will prevent the heartbeat thread from running unless the heartbeat thread is running at a higher priority, and so it will be impossible to receive input keys, etc. (Note that if event collection was in a separate thread it would suffer the same issue; compute intensive code would block the event collection thread unless it was running at higher priority).

So I contrived the experiment below. I'm not sure it properly represents what is happening with the threaded heartbeat, but it still may make interesting discussion.

$vi dynamicPriorityHeartbeat.c

#include <stdio.h> #include <sys/time.h> #include <stdlib.h> #include <time.h> #include <pthread.h>

struct timespec programStartTime; int heartbeat, heartbeatCount;

double elapsed(struct timespec start, struct timespec end) { struct timespec temp; double elapsed; temp.tv_sec = end.tv_sec-start.tv_sec; temp.tv_nsec = end.tv_nsec-start.tv_nsec; if (temp.tv_nsec < 0) { temp.tv_nsec += 1000 * 1000 * 1000; temp.tv_sec -= 1; } elapsed = temp.tv_nsec / 1000 / 1000 ; elapsed = temp.tv_sec + elapsed / 1000; return elapsed; }

void *heart() { int i; for(i=0; i<=heartbeatCount; i++) { printf("Heartbeat %02d ", i); heartbeat=1; usleep(500000); } heartbeat=0; exit(0); }

void runSmalltalk() { struct timespec heartbeatTime; double intenseComputation;

intenseComputation=0;
while(1)
{
    if(!heartbeat)
    {
        intenseComputation += 1;
    }
    else
    {
        heartbeat=0;
        clock_gettime(CLOCK_REALTIME, &heartbeatTime);
        printf("woke at time %f ", elapsed(programStartTime,

heartbeatTime)); printf("intenseComputation=%f\n", intenseComputation); } } }

int main(int argc, char *argv[]) { clock_gettime( CLOCK_REALTIME, &programStartTime); if(argc<2) { printf("Usage: %s heartbeatCount\n", argv[0]); exit(1); } heartbeatCount = atoi(argv[1]); heartbeat=0;

pthread_t heartbeatThread;
if(pthread_create(&heartbeatThread, NULL, heart, NULL))
{
    fprintf(stderr, "Error creating thread\n");
    return 1;
}

runSmalltalk();

} /////////////////////////////////////////

$ gcc dynamicPriorityHeartbeat.c -lpthread && ./a.out 50

Heartbeat 00 woke at time 0.000000 intenseComputation=19006.000000 Heartbeat 01 woke at time 0.500000 intenseComputation=124465517.000000 Heartbeat 02 woke at time 1.000000 intenseComputation=248758350.000000 Heartbeat 03 woke at time 1.500000 intenseComputation=373112100.000000 Heartbeat 04 woke at time 2.000000 intenseComputation=495566843.000000 Heartbeat 05 woke at time 2.500000 intenseComputation=619276640.000000 .... Heartbeat 45 woke at time 22.503000 intenseComputation=5583834266.000000 Heartbeat 46 woke at time 23.003000 intenseComputation=5708079390 <(570)%20807-9390>.000000 Heartbeat 47 woke at time 23.503000 intenseComputation=5831910783.000000 Heartbeat 48 woke at time 24.003000 intenseComputation=5955439495.000000 Heartbeat 49 woke at time 24.503000 intenseComputation=6078752118 <(607)%20875-2118>.000000 Heartbeat 50 woke at time 25.003000 intenseComputation=6202527610 <(620)%20252-7610>.000000

Those times a quite regular. A drift of 3 milliseconds over 25 seconds (0.012%). It seems the intense computation in the main thread does not block timely events in the heatbeat thread.

Perhaps the heatbeat thread needed it static priority managed manually for with Linux 2.4, but maybe that is not required in 2.6 ??

I wonder what happens if and when the main thread starts idling? Then there's the concern that while it *may* just happen to work in particular schedulers it won't work if there's a scheduler change. It seems to me that the safe thing is to use priorities and be sure. But you should easily be able to test the scheme above by modifying the VM to create a heartbeat thread at the same priority as the main thread and then give it some effectively infinite computation that won't cause an interrupt by itself and see if you can interrupt it. e.g.

SmallInteger minVal to: SmallInteger maxVal do: [:i| | sum | sum := 0. 0 to: SmallInteger maxVal // 2 do: [:j| sum := j even ifTrue: [sum + j] ifFalse: [sum - j]]]

This loop shouldn't do any allocations so shouldn't cause a scavenging GC, which would cause an interrupt check. So if you can break into this one in your modified VM then I'd say it was worth experimenting further with your scheme.

cheers -ben

_,,,^..^,,,_ best, Eliot

On Mon, Mar 20, 2017 at 09:07:16PM +0100, Levente Uzonyi wrote:

IMO the best solution would be to add command line parameters to set the priorities - both main and heartbeat using relative values, and keep the current behavior if none was given. Ben's example would be 0 for both parameters. Then the error message on startup could just say to set one of the parameters to run the VM (e.g. 0,0 or -1,0, etc).

A much simpler solution is to just comment out the "exit(errno)" at line 332 of sqUnixHearbeat.c. The resulting VM will work fine.

If the /etc/security/limits.d/squeak.conf is not in place, then you will see the annoying but otherwise harmless warming:

pthread_setschedparam failed: Operation not permitted This VM uses a thread heartbeat who requires a special configuration to work. You need to allow it to run higher priority threads (real time), to allow clock to work properly You need to add a conf file to /etc/security/limits.d, executing this:

sudo cat >/etc/security/limits.d/squeak.conf <<END * hard rtprio 2 * soft rtprio 2 END

Ben's analysis may or may not be exactly right for all conditions of load, but directionally he is right on target. Clock jitter is of little or no interest to the general user, and the people who do care about it are fully capable of setting the necessary security configuration.

AFAIK you can always decrease the priority of the threads on linux, so 0 and negative values should always work.

I think this is correct, so another resonable strategy would be to lower the priority of threads other than the heartbeat thread when the heartbeat priority cannot be raised. But if Ben's analysis is generally correct, then it may not even be worth the trouble of doing that.

Dave

On Tue, Mar 21, 2017 at 1:44 AM, Eliot Miranda eliot.miranda@gmail.com wrote:

Hi David,

On Mon, Mar 20, 2017 at 5:00 PM, David T. Lewis lewis@mail.msen.com wrote:

...

On Mon, Mar 20, 2017 at 09:07:16PM +0100, Levente Uzonyi wrote:

...

IMO the best solution would be to add command line parameters to set the priorities - both main and heartbeat using relative values, and keep the current behavior if none was given. Ben's example would be 0 for both parameters. Then the error message on startup could just say to set one of the parameters to run the VM (e.g. 0,0 or -1,0, etc).

A much simpler solution is to just comment out the "exit(errno)" at line 332 of sqUnixHearbeat.c. The resulting VM will work fine.

Feel free! Do it! Just add a comment to the source.

+1

- Bert -

On Tue, Mar 21, 2017 at 6:18 AM, Holger Freyther holger@freyther.de wrote:

...

On 20 Mar 2017, at 17:43, Ben Coman btc@openinworld.com wrote:

...

...

On Sat, Jan 7, 2017 at 3:23 AM, Eliot Miranda eliot.miranda@gmail.com wrote: This is because its job is to cause Smalltalk code to break out at regular intervals to check for events. If the Smalltalk code is compute-intensive then it will prevent the heartbeat thread from running unless the heartbeat thread is running at a higher priority, and so it will be impossible to receive input keys, etc. (Note that if event collection was in a separate thread it would suffer the same issue; compute intensive code would block the event collection thread unless it was running at higher priority).

...

Perhaps the heatbeat thread needed it static priority managed manually for with Linux 2.4, but maybe that is not required in 2.6 ??

The "flaw" in this experiment is that number of runnable threads is most likely smaller or equal to the number of physical processors. You can use cpu affinity to bind your application to a single CPU and see the delay. I think the "jitter" will likely be higher.

Great insight. I'll try two things: cpu affinity & loading the other cpus with my previous fibinachi experiment.

Keep in mind though that most modern desktop systems are overpowered, usual peaking loads less than 40% unless something goes haywire. And smaller systems like the Pi seem to default to operating as root, so can bump the heatbeat priority.

cheers -ben

On 10-01-2017, at 7:21 AM, Guillermo Polito guillermopolito@gmail.com wrote:

I'd prefer to have a linux VM with a decent default (let's say the itimer one to not bother beginners)

All my experience says that the thread timer *is* a decent default. How many systems are running pre 2.6 level kernels? And why? That’s years out of date.

tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Useful Latin Phrases:- Braccae illae virides cum subucula rosea et tunica Caledonia-quam elenganter concinnatur! = Those green pants go so well with that pink shirt and the plaid jacket!

On 28 Mar 2017, at 17:05, Ben Coman btc@openinworld.com wrote:

Just curious, what is the root cause of this itimer conflict? Is it that a SIGALARM in particular is issued, or just that the current execution is pre-empted to handle the signal - and is that a timing issue, or a concurrency problem where some state is invalidated?

Would it help if to handle the signal in another thread?

AFAICT the main issue is that it can not be shared/multiplexed. E.g. if both ALSA and the VM install an event handler for the signal it is not clear who will win. And they will probably cancel each others work.

Hi Holger,

On Mar 28, 2017, at 9:46 AM, Holger Freyther holger@freyther.de wrote:

...

On 28 Mar 2017, at 17:05, Ben Coman btc@openinworld.com wrote:

Just curious, what is the root cause of this itimer conflict? Is it that a SIGALARM in particular is issued, or just that the current execution is pre-empted to handle the signal - and is that a timing issue, or a concurrency problem where some state is invalidated?

Would it help if to handle the signal in another thread?

AFAICT the main issue is that it can not be shared/multiplexed. E.g. if both ALSA and the VM install an event handler for the signal it is not clear who will win. And they will probably cancel each others work.

Not quite. The problem with ALSA is that it does not follow the convention for installing signal handlers so that they can be chained, nor uninstalling signal handlers so that the previous chain is restored on uninstall.

The main problem is that the SIGALRM will interrupt system calls and it is common to find libraries either poorly written or poorly tested (perfectly understandably) so that the interruptions mean they effectively crash.

ALSA is also in this category.

Hi Ben,

On Mar 28, 2017, at 10:35 AM, Ben Coman btc@openinworld.com wrote:

...

On Tue, Mar 28, 2017 at 11:05 PM, Ben Coman btc@openinworld.com wrote:

...

On Sat, Jan 7, 2017 at 1:33 AM, Eliot Miranda eliot.miranda@gmail.com wrote:

Hi Guille,

...

On Jan 6, 2017, at 6:44 AM, Guillermo Polito guillermopolito@gmail.com wrote:

Hi,

I was checking the code in sqUnixHeartbeat.c to see how the heartbeat thread/itimer worked. It somehow bothers me that there are different compiled artifacts, one per option.

What do you think about having a VM that manages that as an argument provided when we launch the VM? This would add some flexibility that we don't have right now because we make the decision at compile time.

I think it's a fine idea but it isn't really the issue. The issue is that the itimer mechanism is problematic, especially for foreign code, and is therefore a stop gap. The itimer interrupts long-running system calls, which means that things like sound libraries break (at Qwaq I had to fix ALSA to get it to work with the itimer heartbeat). Since Pharo is becoming more reliant on external code it may impact us more going forward.

Just curious, what is the root cause of this itimer conflict? Is it that a SIGALARM in particular is issued, or just that the current execution is pre-empted to handle the signal - and is that a timing issue, or a concurrency problem where some state is invalidated?

Would it help if to handle the signal in another thread?

bah, sorry, accidental bump sent too early. To continue..

POSIX timer_create() & timer_settime() might be worth considering, which can specify a different signal than SIGALARM, and also can apparently send the signal to a specific thread with with SIGEV_THREAD_ID https://linux.die.net/man/2/timer_create https://lists.gt.net/linux/kernel/1289398

Now when some places say there can only be one signal handler per signal per process, but IIUC modern Linux there is no distinction between process or thread, so its one signal handler per signal per thread.

Or perhaps you just need a particular callback function called when the timer expires, per Patryk's answer... http://stackoverflow.com/questions/5740954/problem-in-timers-and-signal Although creating a new thread each time may be too much overhead.

btw, did the change in Linux 2.6.12 from setitimer signalling individual threads to signalling only the main thread have any noticeable effect on the operation of the VM and external code like OSProcess and libraries? per answer by osgx... http://stackoverflow.com/questions/2586926/setitimer-sigalrm-multithread-pro...

I don't know. I do not ha e the cycles or the inclination to keep up with Linux kernel changes. Once something works I tend to move on.

I have noticed (but have to test more carefully) that the VM as ammended by David Lewis not to abort if permission to raise thread priority is denied ran on my 6.5 CentOS installation without complaint after I renamed squeak.conf, logged out and logged back in. I need the check that this continues after a reboot rather than just log out/log in. But perhaps the thread priority issue is fixed.

Anyway, your interest and expertise is much appreciated. Feel free to experiment and make changes. Realize that I'm not "all over this" issue because there are many fish to fry. So I'm very thankful for your efforts here.

cheers -ben

ahh, just saw this from Holger...

...

AFAICT the main issue is that it can not be shared/multiplexed. E.g. if both ALSA and the VM install an event handler for the signal it is not clear who will win. And they will probably cancel each others work.

So I guess that means ALSA is using SIGALRM? Since that is what setitmer is hardcoded to send. I see a few mentions here... http://git.alsa-project.org/?p=alsa-kernel.git&a=search&h=HEAD&s...

cheers -ben

On Wed, Apr 12, 2017 at 9:18 AM, Eliot Miranda eliot.miranda@gmail.com wrote:

Hi Ben,

On Sat, Apr 8, 2017 at 6:39 PM, Ben Coman btc@openinworld.com wrote:

...

On Fri, Jan 6, 2017 at 10:44 PM, Guillermo Polito guillermopolito@gmail.com wrote:

...

Hi,

I was checking the code in sqUnixHeartbeat.c to see how the heartbeat thread/itimer worked. It somehow bothers me that there are different compiled artifacts, one per option.

What do you think about having a VM that manages that as an argument provided when we launch the VM? This would add some flexibility that we don't have right now because we make the decision at compile time.

Can someone advise how to create a new VM setting and read/write to it from the Image?

Give me an example of the kind of setting and I'll make you through it.

Thanks. I'll ask further when I've got something more concrete to work with.

The kinds of settings I know about are:

1. has a command line keyword and argument to set it

2. has a persistent value stored in the image header and is set via vmParameterAt:[put:]

[2.] is the one I was asking about, but I should also pay attention to [1.].

3. 1 + 2

...

As a challenge, I'd actually like to play with dynamically switching the heatbeat between threaded & timer while the Image is running.

Also it would be nice for the Image to be able to inspect what priorities the heatbeat thread is running at.

Ugh :-( That's a lot of plumbing. Much easier to just put a printf in a custom version of the VM somewhere?

Where's the challenge in that? ;) Whether it should be default part of the VM is different question.

...

As well, what is the simplest reliably crashing example of signal conflict with the timer-beat VM?

There isn't a way that reliably crashes the Vm (it doesn't crash). Instead I posted an example which should demonstrate the problem. Here it is again.

[| infiniteLoop | infiniteLoop := [| sum | sum := 0. [sum < 10] whileTrue: [sum := sum + (sum even ifTrue: [1] ifFalse: [-1])]] newProcess. infiniteLoop resume. Processor activeProcess priority: Processor activePriority + 1. (Delay forSeconds: 1) wait. infiniteLoop terminate. Processor activeProcess priority: Processor activePriority - 1] timeToRun 1001

IIUC, that tests the thread-beat VM. Sorry I switched topics. I was looking for examples where OSProcess(?) or otherwise fails due to (presumed) conflict with the itimer-beat signals.

cheers -ben