[OpenSmalltalk/opensmalltalk-vm] Error message for the failure to create heartbeat thread is too specific. (#540)
Hi,
The warning/error
pthread_setschedparam failed: Not owner
is also happening on Solaris, and obviously it also prints some text, which does not really make sense on Solaris.
I have documented on the following Wiki how to get rid of this warning/error on Solaris:
https://sourceforge.net/p/solaris-squeak/wiki/Home/
I don't know how to fix the issue specifically on your Ubuntu distribution, but possibly it makes sense to - when the error happens - to print a simple:
"error: pthread_setschedparam failed: Not owner
or
"error: heartbeat thread unable to run at higher priority"
Such a simple short message would be sufficient, and then the user can try to do some research and find fixes for the specific platform, where the error happens.
I can imagine that on various Unix flavors the instructions for:
/etc/security/limits.d/squeak.conf
do not apply.
Also note that in my limited experience on Solaris, I can either fix the warning (by having threads run at different priorities) or ignore the warning, as it does not seem to make much difference ...
But it's not nice that an extensive error message is printed, certainly not when it does not apply to that specific Unix distribution.
Regards, David Stes
Hi everyone,
It's good that there is an open issue for this issue.
Also note that there are 2 different things here : (1) the issue itself which is that pthread_setschedparam failed: Not owner and (2) how to solve this
For the message, it would be sufficient to just report (1) what happens.
Now the action that one should take to resolve this, depends on the operating system, and also to some extent I think there may be different approaches.
For example, I have recently discovered that an additional necessary privilege must be given at the Solaris zone level :
example# zonecfg -z myzone zonecfg:myzone> set limitpriv="default,proc_priocntl" zonecfg:myzone2> exit
To be able to add "privs=proc_priocntl" in a zone.
One possible approach to address the issue itself is by creating a profile:
# profiles -p Squeak profiles:Squeak> set desc="Allow Squeak VM to Set Priority" profiles:Squeak> add cmd=/usr/bin/squeak profiles:Squeak:squeak> set privs=proc_priocntl profiles:Squeak:squeak> end profiles:Squeak> commit profiles:Squeak> info name=Squeak desc=Allow Squeak VM to Set Priority cmd=/usr/bin/squeak profiles:Squeak> exit
However bottom line is there is a difference between the issue itself, and the many different ways to address the issue on the various platforms and distributions of operating systems.
Regards, David Stes
Hi David, would it be practicable to create a page on the wiki here in this repository that lists the solutions fir the various platforms and then have the error message in the vm reference that wiki page?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Currently if I go to
http://wiki.squeak.org/squeak/search
and I type in the "Search" field : pthread_setschedparam
There is "No Match".
The recent changes to the Wiki
http://wiki.squeak.org/squeak/recent
seem to be related mostly to programming in Squeak.
But there are also wiki pages on building the VM software, so a document on the wiki about the
"pthread_setschedparam failed: Not owner ... See https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/r3732#linux"
issue could be a good idea on wiki.squeak.org.
David Stes
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Another thing to consider is whether calling pthread_setschedparam() should be the default ...
My impression is that if I just ignore the warning
"pthread_setschedparam failed: Not owner"
the programs still run fine.
So perhaps the default could be changed NOT to issue a pthread_setschedparam() and make it optional.
Maybe some option like '-best-performance' could enable this.
Currently the message seems to indicate it is for "best operation".
So the message implies that the default could be not to issue the pthread_setschedparam() and then have the user it enable via a switch.
In any case, the message is very specific and it combines both error and solution, while the solution is "cookbookish" it tells you how what to do, but there are many variations possible and many operating systems, what could be the right "cookbook recipe" on one system may not apply to another ...
MESSAGE:
pthread_setschedparam failed: Not owner This VM uses a separate heartbeat thread to update its internal clock and handle events. For best operation, this thread should run at a higher priority, however the VM was unable to change the priority. The effect is that heavily loaded systems may experience some latency issues. If this occurs, please create the appropriate configuration file in /etc/security/limits.d/ as shown below:
cat <<END | sudo tee /etc/security/limits.d/squeak.conf * hard rtprio 2 * soft rtprio 2 END
and report to the squeak mailing list whether this improves behaviour.
You will need to log out and log back in for the limits to take effect. For more information please see https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/r3732#linux
David Stes
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
By the way, there seems to be an alternative to running the Squeak VM in a profile which raises the privilege temporarily.
It also seems possible to permanently raise the privilege, for example, in a zone by the /etc/security/policy.conf file.
# grep proc_priocntl /etc/security/policy.conf PRIV_DEFAULT=basic,proc_priocntl
By raising the PRIV_DEFAULT to also allow process/light weight processes (threads) to set their own priority (which is not by default allowed), it is possible to get rid of the pthread_setschedparam() warning.
Now permanently allowing this for all threads is best limited, in a zone (a jail) inside the operating system, by setting the limit of the zone to:
# zonecfg -z newt info | grep limitpriv limitpriv: default,proc_priocntl
David Stes
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
When running the Web Framework 'Aida 6.8' on the Cog VM 5.0, I do notice that there seems to be interest in solving the "heartbeat thread" warning.
If I just ignore the warning about the heartbeat thread priority, small Squeak examples and the Squeak system itself seems to run fine.
However the Aida web framework seems to depend on 'scheduling' issues, and Aida apparently works better if I modify the /etc/security/policy.conf
Or maybe it just appears to work better.
# grep proc_priocntl /etc/security/policy.conf PRIV_DEFAULT=basic,proc_priocntl
In any case modifying the /etc/security/policy.conf gets rid of the warning.
The warning itself is however phrased in such a way that there's no indication, that you have to modify the /etc/security/policy.conf of course.
My guess is that there's a great many choices and different options, on how to allow a process to set its priority, on various systems.
David Stes
Hi David, you say
"Another thing to consider is whether calling pthread_setschedparam() should be the default ...
My impression is that if I just ignore the warning
"pthread_setschedparam failed: Not owner"
the programs still run fine.
So perhaps the default could be changed NOT to issue a pthread_setschedparam() and make it optional."
This is not the right solution, but it's a little non-obvious.
What the setschedparam call does is raise the priority of the heartbeat thread relative to the main Smalltalk thread. Leaving it out means that the heartbeat thread runs at the same priority as the main Smalltalk thread. The effect of this is that if the main Smalltalk thread runs continuously, for example is in some compute-intensive loop, then the heartbeat thread will be prevented from running, and hence events such as Delay expiry will not be checked for, and the main thread will never be interrupted. Try something like the following with a heartbeat thread running at the same priority as the main thread running on a single core machine and you'll manifest the issue:
```| p | p := [| i | i := 0. [(i := i + 1) odd ifTrue: [i := i - 1]] repeat] newProcess. p forkAt: Processor activePriority - 1. (Delay forSeconds: 1) wait. p terminate```
without the heartbeat thread running at a higher priority process p consumes all the processor and the heartbeat thread never gets to run, consequently the Delay forSeconds: 1) wait does not complete and p continues running.
The effect of this is that if the main Smalltalk thread runs continuously, for example is in some compute-intensive loop, then the heartbeat thread will be prevented from running, and hence events such as Delay expiry will not be checked for, and the main thread will never be interrupted.
I doubt any modern OS does cooperative scheduling at any priority level. Without the increased priority, the heartbeat will still be scheduled but it may not interrupt the main thread at the intended time. Also, today's machines are almost all multi-core, so even if you keep a core busy with the main thread, another core may schedule the heartbeat process.
On 2020-12-21, at 3:30 PM, smalltalking notifications@github.com wrote:
The effect of this is that if the main Smalltalk thread runs continuously, for example is in some compute-intensive loop, then the heartbeat thread will be prevented from running, and hence events such as Delay expiry will not be checked for, and the main thread will never be interrupted.
I doubt any modern OS does cooperative scheduling at any priority level. Without the increased priority, the heartbeat will still be scheduled but it may not interrupt the main thread at the intended time. Also, today's machines are almost all multi-core, so even if you keep a core busy with the main thread, another core may schedule the heartbeat process.
Well, yeah, that's really the point - you can't rely upon it. On some machines, with some versions of some OS it may work perfectly. On another day it might not.
The bit that has been bothering me recently with this is that a simple test program to try the priority raise seems to work but the nominally-the-same code in the actual VM tells me the raise failed. I'm not a fan of inconsistency.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim There can never be a computer language in which you cannot write a bad program.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Well I can understand that Eliot has good reasons to raise the priority of the heartbeat thread.
Also I have noticed now that I'm running a nice package "AIDA" (web framework on top of Swazoo) that the WebScheduler is sensitive to scheduling of course.
My feeling about it is that still the error message is "too specific".
Perhaps the "configure" script could define something like
HAVE_LINUX_PAM_LIMITS
because I believe the directory /etc/security/limits.d on Linux is related to a pam_limits package.
The platforms/vm/unix/sqUnixHeartbeat.c could then print the message only
#ifdef HAVE_LINUX_PAM_LIMITS
Some Linux distros may not even have that directory /etc/security/limits.d.
Regarding Solaris, or other Unix systems that are not Linux, they can have their own system of privileges to allow/disallow a thread or a process to set or raise priority.
See https://en.wikipedia.org/wiki/Pluggable_authentication_module
So although Solaris uses PAM, I could then not define HAVE_LINUX_PAM_LIMITS, and then at the least the error message would be more suitable.
Although the action to remediate the issue would be different, the heartbeat thread could still run at a different priority than the main thread.
David Stes
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On some Linux systems without PAM,
it might be a possibility to use "prlimit" to set rtprio:
https://serverfault.com/questions/889635/seting-rtprio-limit-in-system-witho...
For example with
prlimit --rtprio=99 --pid=<pid of shell>
and then start squeak from that shell.
If the Linux system has no /etc/security directory that's a possibility.
Basically I think there are probably a great many different ways to achieve the goal of suppressing the "pthread_setsched" message.
The current error message prints (1) a problem and (2) a Linux PAM specific cookbookish recipe to solve it.
However the error message could be limited to just (1) print the problem, that's all ... The user has then to figure out for themselves how to solve it.
David Stes
vm-dev@lists.squeakfoundation.org
-
dcstes -
Eliot Miranda
-
smalltalking
-
tim Rowledge