If security is the goal, this seems not to be the first place to spend scarce developer time.  

What are the vectors by which an attacker can cause such malicious bytecodes to be executed?  The first three that come to mind are:
- direct access to method dictionaries and/or unrestricted compiler access
- providing malicious input to a system-provided binary code loader
- exploiting bugs in the compiler

If the first attack vector is available, crashes due to malicious bytecodes are the least of your problems; arbitrary code execution is a bigger concern.  Glancing at the SecureSqueak page, it seems like you probably have a plan for this.  Have you already solved this problem?  If not, there's no point in bulletproofing the VM against ill-formed bytecodes.

As I mentioned in response to Mathieu, it seems to me that the second attack vector can mostly be dealt with by load-time inspection.  I'm not intimately familiar with Squeak's bytecodes, but I'd be surprised if there were more than a few where run-time checks are actually required.

The third case assumes that the compiler is restricted in some way (eg: the attacker cannot simply "crash" the system by compiling a method containing "Smalltalk snapshot: false andQuit: true"); instead they have to find a way to write code such that the compiler accidently generates invalid bytecodes.  To provide an extra layer of security, we can always subject the newly-compiled method to the same inspection as we do above when loading binary code.

Does this sound reasonable?


On Dec 28, 2007, at 1:41 PM, Michael van der Gulik wrote:

Hi all.

Is the policy of the VM makers (whoever they currently are) to prevent the VM from crashing, particularly when given malicious bytecodes?

This is a general question, mostly related to http://bugs.squeak.org/view.php?id=1395 which is now closed. Is it considered a bug if I can crash the VM with a maliciously crafted method?

Which direction would the Squeak community want to go in? Should we aim to have a VM that would never seg fault and dump core (or blue screen under Windows), regardless of what rubbish is fed to it? Doing extra sanity checks and bounds checking would possibly have a performance penalty.