Squeak-dev March 2017

squeak-dev@lists.squeakfoundation.org

50 participants
281 discussions

Installing Filetree
by Bert Freudenberg 02 May '17

02 May '17

Hi all, Dale has been working on a Squeak version of FileTree, which is a Monticello repository backed not by zip files, but a tree of individual class files that can be versioned using git, svn, etc: https://github.com/dalehenrich/filetree/blob/master/doc/SqueakInstall.md Since I don't like cluttering my image with unrelated things, here's how to load it using just Installer - provided you have an up-to-date trunk image with the fixes I posted today: ======================== "Bootstrap from squeak43 branch in squeaksource3" Installer ss3 project: 'FileTree'; install: 'MonticelloFileTree-Core.squeak43'. "Load from local filetree repository (cloned squeak4.3 branch from github into image folder)" Installer monticello mc: (MCFileTreeRepository directory: (FileDirectory uri: 'filetree/repository/')); install: 'MonticelloFileTree-Core' ======================== If you have an older image without the new fixes, giving an explicit version number for the ss3 bootstrap should make it work (e.g. 'MonticelloFileTree-Core.squeak43-dkh.70'). - Bert -

9 39

Ma Client Server & Squeak launch issues
by Hari Balaraman 20 Apr '17

20 Apr '17

Hi All, I've been attempting to use the Ma Client Server package on Squeak 5.0 using the instructions on http://wiki.squeak.org/squeak/2978 <http://wiki.squeak.org/squeak/2978>. The instructions there appear to be out of date. Is there another source of documentation available? The wiki lists the last edit date of that page in Sep 2016. So I assume the package is still being maintained. Please let me know if that is not the case. Also, the version of Squeak 5.1 on the Squeak.org <http://squeak.org/> does not launch on MacOS. The Squeak 5.0 image provided on Aida/web site does, which I am using. Thanks in advance. Kind regards, Hari

14 28

Crashes on snapshot with the new compactor
by Eliot Miranda 17 Apr '17

17 Apr '17

Hi All, a number of people are being affected by crashes on snapshotting the image, the worst possible time for a crash. There is a bug in the new compactor that unfortunately bites when saving. The compactor is invoked as part of a full garbage collect after the garbage collector has feed unreachable objects. Normally the new compactor makes only a single pass through the heap, which may not move all the objects that are possible to move. (The amount of objects that can be moved in a single pass is limited by available free space.) But on snapshot the compactor makes as may passes as are necessary to slide all movable objects down as far as possible. Unfortunately there is a bug in this second pass. Fixing this bug is now my priority. I have an example image from Esteban Lorenzano to test. I am asking anyone else that can provide an image that reliably crashes when trying to save it to make the image and changes available to me for testing if possible. In the mean time one may be able to work around the problem by doing a full garbage collect before snapshot. This should do a GC with a single compaction pass which should not fail, and then make it much more likely that the GC during snapshot will do a single compaction pass, since fewer objects should be mobile after the single pass compaction in the explicit GC. To do this in Pharo I would put a full gc here: SessionManager>>snapshot: save andQuit: quit | isImageStarting snapshotResult | ChangesLog default logSnapshot: save andQuit: quit. >> SmalltalkImage current primitiveGarbageCollect. self currentSession stop: quit. "Image not usable from here until the session is restarted!" ... In Squeak I would put a full GC here: snapshot: save andQuit: quit withExitCode: exitCode embedded: embeddedFlag "Mark the changes file and close all files as part of #processShutdownList. If save is true, save the current state of this Smalltalk in the image file. If quit is true, then exit to the outer OS shell. If exitCode is not nil, then use it as exit code. The latter part of this method runs when resuming a previously saved image. This resume logic checks for a document file to process when starting up." | resuming msg | Object flushDependents. Object flushEvents. ... Smalltalk processShutDownList: quit. >> SmalltalkImage current primitiveGarbageCollect. Cursor write show. save ifTrue: [resuming := embeddedFlag ifTrue: [self snapshotEmbeddedPrimitive] ifFalse: [self snapshotPrimitive]] "<-- PC frozen here on image file" ifFalse: [resuming := false]. I do apologise for the bug. I hope it will be fixed within a few days. _,,,^..^,,,_ best, Eliot

6 9

Re: The Trunk: Morphic-topa.784.mcz
by Marcel Taeumel 10 Apr '17

10 Apr '17

Nice! :) <http://forum.world.st/file/n4813138/font-importer.png> Two questions: 1. Who does the rendering? Looks kind of irregular... See attachment. 2. Can we move it to the "Tools" package? Best, Marcel -- View this message in context: http://forum.world.st/The-Trunk-Morphic-topa-784-mcz-tp4813130p4813138.html Sent from the Squeak - Dev mailing list archive at Nabble.com.

4 14

The Trunk: XML-Parser-ul.43.mcz
by commits＠source.squeak.org 09 Apr '17

09 Apr '17

Levente Uzonyi uploaded a new version of XML-Parser to project The Trunk: http://source.squeak.org/trunk/XML-Parser-ul.43.mcz ==================== Summary ==================== Name: XML-Parser-ul.43 Author: ul Time: 19 March 2017, 9:24:35.17355 pm UUID: c424f066-2c05-4c84-ab09-71c5099790b9 Ancestors: XML-Parser-dtl.42 Various optimizations to XMLTokenizer and SAXDriver. The most notable change is that the attributes of xml nodes will be set to nil instead of a larger empty Dictionary when they have no attributes. But since its accessor will initialize it to an empty dictionary anyway, we can safely save initializing it while parsing the document. =============== Diff against XML-Parser-dtl.42 =============== Item was changed: ----- Method: SAXDriver>>handleStartTag:attributes:namespaces: (in category 'handling tokens') ----- handleStartTag: elementName attributes: attributeList namespaces: namespaces | localName namespace namespaceURI | + attributeList ifNotNil: [ + (attributeList includesKey: 'xml:lang') ifTrue: [ + languageEnvironment := LanguageEnvironment localeID: (LocaleID isoString: (attributeList at: 'xml:lang')) ] ]. + self usesNamespaces ifFalse: [ + "call the handler" + self saxHandler + checkEOD; + startElement: elementName namespaceURI: nil namespace: nil attributeList: attributeList. + ^self ]. + + self scope enterScope. + "declare any namespaces" + namespaces ifNotNil: [ + namespaces keysAndValuesDo: [:ns :uri | + self scope declareNamespace: ns uri: uri] ]. - (attributeList includesKey: 'xml:lang') - ifTrue: [languageEnvironment := LanguageEnvironment localeID: (LocaleID isoString: (attributeList at: 'xml:lang'))]. - self usesNamespaces - ifTrue: [ - self scope enterScope. - "declare any namespaces" - namespaces keysAndValuesDo: [:ns :uri | - self scope declareNamespace: ns uri: uri]. + self splitName: elementName into: [:ns :ln | + namespace := ns. + localName := ln]. - self splitName: elementName into: [:ns :ln | - namespace := ns. - localName := ln]. + namespaceURI := self scope + namespaceURIOf: namespace + ifAbsent: [self parseError: 'Start tag ' , elementName , ' refers to undefined namespace ' , namespace asString]. - namespaceURI := self scope - namespaceURIOf: namespace - ifAbsent: [self parseError: 'Start tag ' , elementName , ' refers to undefined namespace ' , namespace asString]. + attributeList ifNotNil: [ + self validatesAttributes ifTrue: [ + self scope validateAttributes: attributeList ] ]. + "call the handler" + self saxHandler + checkEOD; + startElement: localName namespaceURI: namespaceURI namespace: namespace attributeList: attributeList! - self validatesAttributes - ifTrue: [self scope validateAttributes: attributeList]. - "call the handler" - self saxHandler - checkEOD; - startElement: localName namespaceURI: namespaceURI namespace: namespace attributeList: attributeList] - ifFalse: [ - "call the handler" - self saxHandler - checkEOD; - startElement: elementName namespaceURI: nil namespace: nil attributeList: attributeList]! Item was changed: Object subclass: #XMLTokenizer + instanceVariableNames: 'stream nestedStreams entities externalEntities parameterEntities parsingMarkup markedPosition peekChar validating stringBuffer stringBufferStack' + classVariableNames: 'CharEscapes DigitTable LiteralChars NameDelimiters' - instanceVariableNames: 'stream nestedStreams entities externalEntities parameterEntities parsingMarkup markedPosition peekChar validating nameBuffer attributeBuffer' - classVariableNames: 'CharEscapes DigitTable LiteralChars NameDelimiters SeparatorTable' poolDictionaries: '' category: 'XML-Parser'! !XMLTokenizer commentStamp: '<historical>' prior: 0! XMLTokenizer bolot(a)cc.gatech.edu breaks the stream of characters into a stream of XMLnodes (aka token stream) token stream is used by XMLparser to generate XMLdocument tree! Item was changed: ----- Method: XMLTokenizer class>>initialize (in category 'class initialization') ----- initialize "XMLTokenizer initialize" CharEscapes := CharacterSet newFrom: #( $& $" $' $> $< ). - SeparatorTable := CharacterSet new. - #(9 10 12 13 32) do: [:each | SeparatorTable add: each asCharacter]. - LiteralChars := CharacterSet newFrom: #( $: $- $_ $= $.). 0 to: 255 do: [:i | | char | char := i asCharacter. (char isDigit or: [char isLetter]) ifTrue: [LiteralChars add: char]]. NameDelimiters := CharacterSet new. #(9 10 12 13 32 61 "$= asInteger 61" 62 "$> asInteger" 47 "$/ asInteger") do: [:each | NameDelimiters add: each asCharacter]. DigitTable := Array new: 256. DigitTable atAllPut: -1. ($0 to: $9) do: [:each | DigitTable at: each asciiValue put: each digitValue]. ($a to: $f) do: [:each | DigitTable at: each asciiValue put: each digitValue]. ($A to: $F) do: [:each | DigitTable at: each asciiValue put: each digitValue]. ! Item was changed: ----- Method: XMLTokenizer>>atEnd (in category 'streaming') ----- atEnd + + nestedStreams ifNil: [ + peekChar ifNotNil: [ ^false ]. + ^stream atEnd ]. + stream atEnd ifFalse: [ ^false ]. + ^self + popNestingLevel; + atEnd! - nestedStreams == nil - ifTrue: [^peekChar == nil and: [stream atEnd]]. - ^stream atEnd - ifTrue: [ - self popNestingLevel. - self atEnd] - ifFalse: [false]! Item was removed: - ----- Method: XMLTokenizer>>fastStreamStringContents: (in category 'private') ----- - fastStreamStringContents: writeStream - | newSize | - newSize := writeStream position. - ^(String new: newSize) - replaceFrom: 1 - to: newSize - with: writeStream originalContents - startingAt: 1! Item was changed: ----- Method: XMLTokenizer>>initialize (in category 'initialize') ----- initialize parsingMarkup := false. validating := false. + stringBuffer := (String new: 128) writeStream. + stringBufferStack := OrderedCollection new! - attributeBuffer := WriteStream on: (String new: 128). - nameBuffer := WriteStream on: (String new: 128)! Item was changed: ----- Method: XMLTokenizer>>next (in category 'streaming') ----- next "Return the next character from the current input stream. If the current stream is at end pop to next nesting level if there is one. Due to the potential nesting of original document, included documents and replacment texts the streams are held in a stack representing the nested streams. The current stream is the top one." + | nextChar | + peekChar ifNotNil: [ + nextChar := peekChar. + peekChar := nil. + ^nextChar ]. + nestedStreams ifNotNil: [ self checkNestedStream ]. + ^stream next! - peekChar - ifNil: [ - nestedStreams ifNotNil: [self checkNestedStream]. - ^nextChar := stream next] - ifNotNil: [ - nextChar := peekChar. - peekChar := nil. - ^nextChar]. - ! Item was changed: ----- Method: XMLTokenizer>>nextAttributeValue (in category 'tokenizing') ----- nextAttributeValue + + | delimiterChar nextChar nextPeek referenceString entity entityValue | - | delimiterChar attributeValueStream nextChar nextPeek referenceString entity entityValue | delimiterChar := self next. (delimiterChar == $" or: [delimiterChar == $']) ifFalse: [self errorExpected: 'Attribute value delimiter expected.']. + self pushNewStringBuffer. - attributeValueStream := attributeBuffer reset. [ nextPeek := nextChar := self next. nextChar ifNil: [self errorExpected: 'Character expected.']. nextChar == $& ifTrue: [ self peek == $# ifTrue: [ nextPeek := nil. nextChar := self nextCharReference] ifFalse: [ referenceString := self nextLiteral. self next == $; ifFalse: [self errorExpected: ';']. entity := self entity: referenceString. entityValue := entity valueForContext: #content. (self class isCharEscape: entityValue) ifTrue: [ nextPeek := nil. nextChar := entityValue first] ifFalse: [ entityValue := entityValue asString. entityValue isEmpty ifTrue: [nextPeek := nextChar := nil] ifFalse: [ self pushStream: (ReadStream on: entityValue asString). nextPeek := nextChar := self next]]]]. nextPeek == delimiterChar] whileFalse: [ + nextChar ifNotNil: [stringBuffer nextPut: nextChar]]. + ^self popStringBuffer! - nextChar ifNotNil: [attributeValueStream nextPut: nextChar]]. - ^self fastStreamStringContents: attributeValueStream - " ^attributeValueStream contents"! Item was changed: ----- Method: XMLTokenizer>>nextEntity (in category 'tokenizing') ----- nextEntity "return the next XMLnode, or nil if there are no more. Fixed to retain leading whitespace when PCDATA is detected." |whitespace| "branch, depending on what the first character is" whitespace := self nextWhitespace. self atEnd ifTrue: [self handleEndDocument. ^ nil]. self checkAndExpandReference: (self parsingMarkup ifTrue: [#dtd] ifFalse: [#content]). + self peek == $< ifTrue: [ ^self nextNode ]. + whitespace isEmpty ifFalse: [ self pushBack: whitespace ]. + ^self nextPCData! - ^self peek = $< - ifTrue: [self nextNode] - ifFalse: [whitespace isEmpty - ifFalse: [self pushBack: whitespace]. - self nextPCData]! Item was changed: ----- Method: XMLTokenizer>>nextName (in category 'tokenizing') ----- nextName + + self pushNewStringBuffer. + self peek == $. ifTrue: [ self malformedError: 'Character expected.' ]. + [ + self peek ifNil: [ self errorExpected: 'Character expected.' ]. + NameDelimiters includes: peekChar ] whileFalse: [ + stringBuffer nextPut: self next ]. + ^self popStringBuffer! - | nextChar | - nameBuffer reset. - self peek == $. - ifTrue: [self malformedError: 'Character expected.']. - [(nextChar := self peek) - ifNil: [self errorExpected: 'Character expected.']. - NameDelimiters includes: nextChar] whileFalse: [ - nameBuffer nextPut: self next]. - ^self fastStreamStringContents: nameBuffer! Item was changed: ----- Method: XMLTokenizer>>nextNode (in category 'tokenizing') ----- nextNode | nextChar | "Skip < " self next. + self peek == $!! ifTrue: [ - nextChar := self peek. - nextChar == $!! ifTrue: [ "Skip !!" self next. nextChar := self peek. nextChar == $- ifTrue: [^self nextComment]. nextChar == $[ ifTrue: [^self nextCDataOrConditional]. + self parsingMarkup ifTrue: [ ^self nextMarkupDeclaration ]. + ^self nextDocType ]. + peekChar == $? ifTrue: [^self nextPI]. - ^self parsingMarkup - ifTrue: [self nextMarkupDeclaration] - ifFalse: [self nextDocType]]. - nextChar == $? ifTrue: [^self nextPI]. ^self nextTag! Item was changed: ----- Method: XMLTokenizer>>nextPCData (in category 'tokenizing') ----- nextPCData - | resultStream nextChar referenceString entity entityValue nextPeek | - resultStream := (String new: 10) writeStream. - self validating - ifFalse: [ - [self peek == $<] - whileFalse: [resultStream nextPut: self next]. - ^self handlePCData: resultStream contents]. + | nextChar referenceString entity entityValue nextPeek | + self validating ifFalse: [ + self peek == $< ifTrue: [ ^self handlePCData: '' ]. + self pushNewStringBuffer. + [ self peek == $< ] + whileFalse: [ stringBuffer nextPut: self next ]. + ^self handlePCData: self popStringBuffer ]. + + self pushNewStringBuffer. [ nextPeek := nextChar := self peek. nextChar ifNil: [self errorExpected: 'Character expected.']. nextChar == $& ifTrue: [ self next. self peek == $# ifTrue: [ nextPeek := nil. nextChar := self nextCharReference] ifFalse: [ referenceString := self nextLiteral. self next == $; ifFalse: [self errorExpected: ';']. entity := self entity: referenceString. entityValue := entity valueForContext: #content. (self class isCharEscape: entityValue) ifTrue: [ nextPeek := nil. nextChar := entityValue first] ifFalse: [ entityValue := entityValue asString. entityValue isEmpty ifTrue: [nextPeek := nextChar := nil] ifFalse: [ + self pushStream: entityValue readStream. - self pushStream: (ReadStream on: entityValue asString). nextPeek := nextChar := self peek]]]] ifFalse: [nextPeek == $< ifFalse: [self next]]. nextPeek == $<] whileFalse: [ + nextChar ifNotNil: [stringBuffer nextPut: nextChar]]. + self handlePCData: self popStringBuffer! - nextChar ifNotNil: [resultStream nextPut: nextChar]]. - self handlePCData: resultStream contents! Item was changed: ----- Method: XMLTokenizer>>nextTag (in category 'tokenizing') ----- nextTag + + | tagName attributes namespaces | + self peek == $/ ifTrue: [^self nextEndTag]. - | tagName attributes nextChar namespaces | - (self peek = $/) - ifTrue: [^self nextEndTag]. tagName := self nextName. self skipSeparators. + attributes := nil. + namespaces := nil. + [ self peek == $> or: [ peekChar == $/ ] ] whileFalse: [ + self + checkAndExpandReference: #content; + nextAttributeInto: (attributes ifNil: [ attributes := Dictionary new ]) + namespaces: (namespaces ifNil: [ namespaces := Dictionary new ]); + skipSeparators ]. - attributes := Dictionary new: 33. - namespaces := Dictionary new: 5. - [(nextChar := self peek) == $> or: [nextChar == $/]] whileFalse: [ - self checkAndExpandReference: #content. - self nextAttributeInto: attributes namespaces: namespaces. - self skipSeparators.]. self handleStartTag: tagName attributes: attributes namespaces: namespaces. + self next == $/ ifTrue: [ + self + handleEndTag: tagName; + next ]! - self next == $/ - ifTrue: [ - self handleEndTag: tagName. - self next]. - ! Item was changed: ----- Method: XMLTokenizer>>nextTrimmedBlanksUpTo: (in category 'streaming') ----- nextTrimmedBlanksUpTo: delimiter + + | nextChar | + self pushNewStringBuffer. - | resultStream nextChar | - resultStream := WriteStream on: (String new: 10). - nextChar := nil. [(nextChar := self next) == delimiter] whileFalse: [ nextChar == $ ifFalse: [ + stringBuffer nextPut: nextChar]]. - resultStream nextPut: nextChar]]. nextChar == delimiter ifFalse: [self parseError: 'XML no delimiting ' , delimiter printString , ' found']. + ^self popStringBuffer - ^resultStream contents ! Item was changed: ----- Method: XMLTokenizer>>nextWhitespace (in category 'tokenizing') ----- nextWhitespace + + | resultString | + "Optimize the most common case away." + self peek ifNil: [ ^'' ]. + peekChar isSeparator ifFalse: [ ^'' ]. + + self pushNewStringBuffer. + [ self peek + ifNil: [ false ] + ifNotNil: [ peekChar isSeparator ] ] + whileTrue: [ stringBuffer nextPut: self next ]. - | nextChar resultStream resultString| - resultStream := (String new: 10) writeStream. - [((nextChar := self peek) ~~ nil) - and: [SeparatorTable includes: nextChar]] - whileTrue: [resultStream nextPut: nextChar. self next]. (nestedStreams == nil or: [self atEnd not]) ifFalse: [self checkNestedStream. self nextWhitespace]. + resultString := self popStringBuffer. - resultString := resultStream contents. resultString isEmpty ifFalse: [self handleWhitespace: resultString]. ^resultString! Item was changed: ----- Method: XMLTokenizer>>peek (in category 'streaming') ----- peek "Return the next character from the current input stream. If the current stream poop to next nesting level if there is one. Due to the potential nesting of original document, included documents and replacment texts the streams are held in a stack representing the nested streams. The current stream is the top one." + + ^peekChar ifNil: [ + nestedStreams ifNotNil: [ self checkNestedStream ]. + peekChar := stream next ]! - peekChar - ifNil: [ - nestedStreams ifNotNil: [self checkNestedStream]. - ^peekChar := stream next] - ifNotNil: [^peekChar]! Item was added: + ----- Method: XMLTokenizer>>popStringBuffer (in category 'private - buffering') ----- + popStringBuffer + + | previousPosition result | + previousPosition := stringBufferStack removeLast. + result := stringBuffer originalContents + copyFrom: previousPosition + 1 + to: stringBuffer position. + stringBuffer position: previousPosition. + ^result! Item was added: + ----- Method: XMLTokenizer>>pushNewStringBuffer (in category 'private - buffering') ----- + pushNewStringBuffer + + stringBufferStack addLast: stringBuffer position + ! Item was changed: ----- Method: XMLTokenizer>>skipSeparators (in category 'streaming') ----- skipSeparators + + [ self peek + ifNil: [ false ] + ifNotNil: [ peekChar isSeparator ] ] + whileTrue: [ self next ]. + nestedStreams ifNil: [ ^self ]. + self atEnd ifTrue: [ + self + checkNestedStream; + skipSeparators ]! - | nextChar | - [((nextChar := self peek) ~~ nil) - and: [SeparatorTable includes: nextChar]] - whileTrue: [self next]. - (nestedStreams == nil or: [self atEnd not]) - ifFalse: [ - self checkNestedStream. - self skipSeparators]!

2 1

Morphs can be dropped on main docking bar
by Bernhard Pieber 07 Apr '17

07 Apr '17

I just found out that I can drop morphs on the main docking bar. When I show the docking bar’s context menu, the „accept drops“ checkbox is not clicked, though. Is this a bug or a feature? Bernhard

10 21

Broken trunk update and workaround
by Eliot Miranda 06 Apr '17

06 Apr '17

Hi All, I'm very sorry to report that my recent changes (adding CompiledCode and CompiledBlock to the CompiledMethod hierarchy in anticipation of full blocks, and merging ContextPart and MethodContext and renaming them as Context, to eliminate obsolete block and closer implementations) has broken the update process. I do not believe there is an easy fix as i think the issues are to do with the citations that exist on the stack while the update proceeds (the progress indicators). There is a clumsy work-around that appears reliable. The pauses between loads get rid of the progress indication activations and hence avoid trying to upgrade running processes through these kernel changes. Open the trunk repository from your Monticello Browser. Locate the update package (the penultimate entry in the list of packages). Load (or merge) update-eem.400.mcm. Load (or merge) update-eem.403.mcm. Load (or merge) update-eem.406.mcm. Save your image. Local the Kernel package. Load Kernel-eem.1078. Save your image. Update I'm so sorry. Eliot

7 24

TimeStamp - can we remove references to this class?
by Chris Cunningham 01 Apr '17

01 Apr '17

Hi. First, the background. I've been living in (and using daily) an image dating back to roughly 4.10 times, but updated to some point post-5.1 , and lately it has become very clear that I have really done something nasty to the image. When I save and quit, errors pop up, but it saves successfully and restarts without errors. On the machine I saved it on. I can't get the image to run on any other system that I have access too, which is scary. I think this is related to the fact that I use ODBC, which has its own definition of TimeStamp. This has collided significantly with the current Squeak definition of TimeStamp, and I think this collision has caused my these issues. I'll be working on a new packaging of the ODBC client that doesn't reference TimeStamp at all, but rather deals with DateAndTime. But that's just one package that used a 'TimeStamp' - I believe there are others out there. Which brings up my question here: Can we remove TimeStamp from the back package? Its representation is definitely not what any older package would expect of TimeStamp. TimeStamp appears to be a shim over DateAndTime for: TestRunner/SUnit, SqueakMap, Monticello, and ChangeRecord. These could learn to use the DateAndTime, although that would require careful changes (very careful!). Thanks, cbc

4 4

The Trunk: Compression-nice.52.mcz
by commits＠source.squeak.org 01 Apr '17

01 Apr '17

Nicolas Cellier uploaded a new version of Compression to project The Trunk: http://source.squeak.org/trunk/Compression-nice.52.mcz ==================== Summary ==================== Name: Compression-nice.52 Author: nice Time: 1 April 2017, 12:30:20.191934 am UUID: 470505e6-d35d-450a-9804-29b904c5853d Ancestors: Compression-ul.51 Correct a bug that can cause ZlibPlugin to read past the zipCollection boundary in DeflateBlock. If the plugin is disabled, there will be a 'subscript is out of bounds: 65537' in DeflateStream fallback code. ByteString(Object)>>error: ByteString(Object)>>errorSubscriptBounds: ByteString(Object)>>at: ByteString>>at: ByteString>>byteAt: GZipWriteStream(DeflateStream)>>updateHashAt: GZipWriteStream(DeflateStream)>>insertStringAt: GZipWriteStream(DeflateStream)>>deflateBlock:chainLength:goodMatch: GZipWriteStream(ZipWriteStream)>>deflateBlock:chainLength:goodMatch: GZipWriteStream(DeflateStream)>>deflateBlock GZipWriteStream(DeflateStream)>>next:putAll:startingAt: GZipWriteStream(DeflateStream)>>nextPutAll: ByteString(String)>>zipped This bug is triggered by some input data, for example if you extract the snapshot.bin file from the zip archive the archive http://source.squeak.org/VMMaker/VMMaker.oscog-nice.794.mcz We need MinMatch bytes to insert the string following the best match. (confirmed by comment near the end of a C version http://git.savannah.gnu.org/cgit/gzip.git/tree/deflate.c) Since we pass lastIndex-1 to the primitive, and that the longest match can be up to MaxMatch, we are going to read the zipCollection up to: lastIndex - 1 + MaxMatch - 1 + MinMatch - 1 (0 based) Thus we must have: lastIndex - 1 + MaxMatch - 1 + MinMatch - 1 < writeLimit We have MinMatch = 3, thus the simplification lastIndex + MaxMatch < writeLimit Thus the requirement: lastIndex + MaxMatch + 1 <= writeLimit =============== Diff against Compression-ul.51 =============== Item was changed: ----- Method: DeflateStream>>deflateBlock (in category 'deflating') ----- deflateBlock "Deflate the current contents of the stream" | flushNeeded lastIndex | (blockStart == nil) ifTrue:[ "One time initialization for the first block" 1 to: MinMatch-1 do:[:i| self updateHashAt: i]. blockStart := 0]. [blockPosition < position] whileTrue:[ + (position + MaxMatch + 1 > writeLimit) + ifTrue:[lastIndex := writeLimit - MaxMatch - 1] - (position + MaxMatch > writeLimit) - ifTrue:[lastIndex := writeLimit - MaxMatch] ifFalse:[lastIndex := position]. flushNeeded := self deflateBlock: lastIndex-1 chainLength: self hashChainLength goodMatch: self goodMatchLength. flushNeeded ifTrue:[ self flushBlock. blockStart := blockPosition]. "Make room for more data" self moveContentsToFront]. !

2 1

Buffer overrun in ZipPlugin
by Nicolas Cellier 01 Apr '17

01 Apr '17

Hi, some remember that from time to time, especially when the ZipPlugin is disabled, the DeflateStream fallback code is failing with a 'subscript is out of bounds: 65537' I have a reproducible example with quite a large file | zip file | zip := (FileStream fileNamed: 'snapshot.bin.gz') binary contentsOfEntireFile asString. file := zip unzipped. file zipped unzipped = file which gives me: ByteString(Object)>>error: ByteString(Object)>>errorSubscriptBounds: ByteString(Object)>>at: ByteString>>at: ByteString>>byteAt: GZipWriteStream(DeflateStream)>>updateHashAt: GZipWriteStream(DeflateStream)>>insertStringAt: GZipWriteStream(DeflateStream)>>deflateBlock:chainLength:goodMatch: GZipWriteStream(ZipWriteStream)>>deflateBlock:chainLength:goodMatch: GZipWriteStream(DeflateStream)>>deflateBlock GZipWriteStream(DeflateStream)>>next:putAll:startingAt: GZipWriteStream(DeflateStream)>>nextPutAll: ByteString(String)>>zipped When the plugin is enabled, the example works. So I've been tracking the differences between DeflatePlugin and DeflateStream fallback code for some times without success. What if I add a bound check in the DeflatePlugin? Then the plugin equally fails. I mean it reads past zipCollection bounds. So there's something bad in the implementation. Why does the example seem to work is still mysterious to me.

1 2

Jump to page: