Squeak-dev December 2015

squeak-dev@lists.squeakfoundation.org

60 participants
161 discussions

Daily Commit Log
by commits＠source.squeak.org 20 Dec '15

20 Dec '15

Changes to Trunk (http://source.squeak.org/trunk.html) in the last 24 hours: http://lists.squeakfoundation.org/pipermail/packages/2015-December/009216.h… Name: Morphic-tpr.1059 Ancestors: Morphic-tpr.1058 When opening sub-menus with insufficient rightwards space on the Display we instead put them to the left of the parent. This change makes the leftward slide a bit smaller so that the mouse stays over menus was the user slides to the new menu; thus no strange closing of menus etc. ============================================= http://lists.squeakfoundation.org/pipermail/packages/2015-December/009217.h… Name: Morphic-tpr.1060 Ancestors: Morphic-tpr.1059, Morphic-mt.1059 Attempt to correct stupid (probably( mistake in MC usage; I used the 'copy image versions here' menu item in the belief it would do something more sensible than simply copy a file. This clashed with marcel's change and so this is an attempt to re-merge and save a proper version. =============================================

1 0

The Trunk: Morphic-tpr.1060.mcz
by commits＠source.squeak.org 20 Dec '15

20 Dec '15

tim Rowledge uploaded a new version of Morphic to project The Trunk: http://source.squeak.org/trunk/Morphic-tpr.1060.mcz ==================== Summary ==================== Name: Morphic-tpr.1060 Author: tpr Time: 20 December 2015, 10:57:42.928 am UUID: c999504e-8f74-4941-a116-911eb62f0c74 Ancestors: Morphic-tpr.1059, Morphic-mt.1059 Attempt to correct stupid (probably( mistake in MC usage; I used the 'copy image versions here' menu item in the belief it would do something more sensible than simply copy a file. This clashed with marcel's change and so this is an attempt to re-merge and save a proper version. =============== Diff against Morphic-tpr.1059 =============== Item was changed: ----- Method: Form>>scaledIntoFormOfSize:smoothing: (in category '*Morphic') ----- scaledIntoFormOfSize: aNumberOrPoint smoothing: factor "Scale and center the receiver into a form of a given size" | extent scaledForm result | extent := aNumberOrPoint asPoint. extent = self extent ifTrue: [^ self copy]. scaledForm := self scaledToSize: extent smoothing: factor. + result := self species extent: extent depth: self depth. - result := self species extent: extent depth: 32. result getCanvas translucentImage: scaledForm at: extent - scaledForm extent // 2. ^ result !

1 0

The Trunk: Morphic-tpr.1059.mcz
by commits＠source.squeak.org 20 Dec '15

20 Dec '15

tim Rowledge uploaded a new version of Morphic to project The Trunk: http://source.squeak.org/trunk/Morphic-tpr.1059.mcz ==================== Summary ==================== Name: Morphic-tpr.1059 Author: tpr Time: 18 December 2015, 4:26:36.049 pm UUID: ce2af9d7-dcb1-4225-8fea-1a0b3ae28ab4 Ancestors: Morphic-tpr.1058 When opening sub-menus with insufficient rightwards space on the Display we instead put them to the left of the parent. This change makes the leftward slide a bit smaller so that the mouse stays over menus was the user slides to the new menu; thus no strange closing of menus etc. =============== Diff against Morphic-tpr.1058 =============== Item was changed: ----- Method: MenuMorph>>popUpAdjacentTo:forHand:from: (in category 'control') ----- popUpAdjacentTo: rightOrLeftPoint forHand: hand from: sourceItem "Present this menu at the given point under control of the given hand." | tryToPlace selectedOffset rightPoint leftPoint | hand world startSteppingSubmorphsOf: self. popUpOwner := sourceItem. self fullBounds. self updateColor. "ensure layout is current" selectedOffset := (selectedItem ifNil: [self items first]) position - self position. tryToPlace := [:where :mustFit | | delta | self position: where - selectedOffset. delta := self boundsInWorld amountToTranslateWithin: sourceItem worldBounds. (delta x = 0 or: [mustFit]) ifTrue: [delta = (0 @ 0) ifFalse: [self position: self position + delta]. sourceItem world addMorphFront: self. ^ self]]. rightPoint := rightOrLeftPoint first + ((self layoutInset + self borderWidth) @ 0). + leftPoint := rightOrLeftPoint last + ((self layoutInset + self borderWidth - self width) @ 0). - leftPoint := rightOrLeftPoint last - ((self layoutInset + self borderWidth + self width) @ 0). tryToPlace value: rightPoint value: false; value: leftPoint value: false; value: rightPoint value: true.!

1 0

Re: [Vm-dev] Re: SecureSession frame types
by Robert Withers 20 Dec '15

20 Dec '15

Good morning, just a few comments and questions.. 1) current 4bits for msgVersion and 4bits for hdrType. With 14 messages and some not implemented (suspend/resume/...) we have little headroom in 4bits of hdrType. Shall I make the msgVersion 3bits and the hdrType 5bits? 2) the smallest interleaved FEC encoded msg is 30 bytes. With 8 bytes of data: a msgSpec, using RS(15,9) with 4bit symbols, and 4 symbol block interleaving, the smallest on the wire is 30 bytes. Without FEC enabled is it 8 bytes. 3) when run to the end of hdrType room, we can add a jump tag and have a second hdrType field as the 9th byte. Again, the valid document link is: https://www.dropbox.com/s/zj2i1votg432b2p/FrameTypes.pptx?dl=0 Thank you, robert On 12/20/2015 06:20 AM, Clément Bera wrote: > > > > Hello, > > I tried to get your document but I got: > > Dropbox - error > > Is it only me ? > > 2015-12-20 11:31 GMT+01:00 Robert Withers <robert.w.withers(a)gmail.com > <mailto:robert.w.withers@gmail.com>>: > > > Excuse me for failing to complete this email./ I wanted to ask for > feedback on these structures, especially the header specification > word structure. So these fields make sense? > > thank you, > robert > > > On 12/20/2015 05:27 AM, Robert Withers wrote: > > I wrote up the various SecureSession frame types in > presentation document, which you can get here: > https://www.dropbox.com/s/3xfa5oqs8ihj88v/FrameTypes.pptx?dl=0 > > These are not yet implemented, but it is progressing. Ne quid > nemis. > > thanks, > > > -- > . .. .. ^,^ robert > > -- . .. .. ^,^ robert

1 2

Re: The Inbox: Morphic-tpr.1059.mcz
by marcel.taeumel 20 Dec '15

20 Dec '15

+1 Marcel -- View this message in context: http://forum.world.st/The-Inbox-Morphic-tpr-1059-mcz-tp4867870p4868044.html Sent from the Squeak - Dev mailing list archive at Nabble.com.

1 0

CurrentReadOnlySourceFiles annoys me ...
by marcel.taeumel 19 Dec '15

19 Dec '15

Hi there! The following snippet does not work: [(Morph >> #submorphCount) getSource] on: Exception do: [:ex | ] Because "CurrentReadOnlySourceFiles" is a subclass of Exception. This smells. Does somebody know, why? Rationale: I want to catch all exceptions in some silently but frequently called method. Best, Marcel -- View this message in context: http://forum.world.st/CurrentReadOnlySourceFiles-annoys-me-tp4867253.html Sent from the Squeak - Dev mailing list archive at Nabble.com.

7 18

MenuMorph>popUpAdjacent... seems to mis-position sub-menus
by tim Rowledge 19 Dec '15

19 Dec '15

When a menu has sub-menus they are placed to the right of the parent but slid back a little left so the overlap and the pointer travelling rightwards to the new menu will not suddenly trigger mouseleave vents and close the menu tree unexpectedly. If however the parent menu is nearly at the right side of the screen, the sub-menu has to be placed to the left. So far as I can work out the current code gets this slightly wrong and places it too far left so that the mouse does indeed tend to send a mouseleave event etc. Result is very annoying for users since they can’t select what they wanted to. The culprit appears to me to be in the MenuMorph>popUpAdjacentTo:forHand:from: method where ‘leftPoint’ is calculated. We subtract the layoutinset and borderwidth as well as the width; whereas I’m fairly sure it ought to be *add* the layoutinset and borderwidth but subtract the width. Certainly it seems to provide a better visual layout. I can’t see any cases that get broken by such a change but since it’s hard to be sure I’ll just commit this to the inbox for a couple of days before trunking it. tim -- tim Rowledge; tim(a)rowledge.org; http://www.rowledge.org/tim People who deal with bits should expect to get bitten.

2 1

encoding challenge
by Robert Withers 19 Dec '15

19 Dec '15

A core part of communications, whether online or to persistence, is encoding, or marshalling. Let's do an example! Thank you for your thoughtful, earnest participation, as God wills it. Here are 2 methods that encode and decode a DateAndTime object, but rather poorly. I've been going without much sleep for a bit and I can't figure it out. Here is the code in hopes that some will find this an interesting exercise in team and community building. For that intention, as a beginning. What's the best solution? Let's show them how to code when given the space in the environment. Tnks! :) In class ASN1UTCTimeType.. encodeValue: anObject withDERStream: derStream | yy mo dd hh mm ss utcDateTime | yy := anObject year asString copyFrom: 3 to: 4. mo := anObject month asString padded: #left to: 2 with: $0. dd := anObject dayOfMonth asString padded: #left to: 2 with: $0. hh := anObject hour asString padded: #left to: 2 with: $0. mm := anObject minute asString padded: #left to: 2 with: $0. ss := anObject seconds asString padded: #left to: 2 with: $0. utcDateTime := (yy, mo, dd, hh, mm, ss, 'Z') asByteArray. derStream nextPutAll: utcDateTime. decodeValueWithDERStream: derStream length: length | aUTCDateTime | aUTCDateTime := (derStream next: length) asByteArray asString. ^ (DateAndTime readFromString: ( ((aUTCDateTime copyFrom: 1 to: 2) asInteger > 50 ifTrue: ['19'] ifFalse: ['20']), (aUTCDateTime copyFrom: 1 to: 2), '-', (aUTCDateTime copyFrom: 3 to: 4), '-', (aUTCDateTime copyFrom: 5 to: 6), 'T', (aUTCDateTime copyFrom: 7 to: 8), ':', (aUTCDateTime copyFrom: 9 to: 10), ':', (aUTCDateTime copyFrom: 11 to: 12))) offset: 0 -- . .. .. ^,^ robert

2 3

Fwd: Re: [Pharo-users] [squeak-dev] Name change: Mushroom ( was Re: evolutions of squeakelib & crypto)
by Robert Withers 19 Dec '15

19 Dec '15

:) Thank you! -------- Forwarded Message -------- Subject: Re: [Pharo-users] [squeak-dev] Name change: Mushroom ( was Re: evolutions of squeakelib & crypto) Date: Fri, 18 Dec 2015 22:37:00 -0500 From: Robert Withers <robert.w.withers(a)gmail.com> To: pharo-users(a)lists.pharo.org The Ode to CryptoCeps: Bouncer: "No, I'm sorry. This is a restricted area..." *checks around in case there's a problem* Me: " 'cep I got this..." *hands over teensie tiny tasty little mushoom* Note to self: 'did I eat one of those???' *can't seem to recall the particular moment. Oh@* Community: *looking sideways at each other* "What the hell is a mushoom?" Thank You! so much. On 12/18/2015 10:27 PM, Robert Withers wrote: > Ah ha!!! Thank you...how about CryptOCeps? ;) > > On 12/18/2015 10:22 PM, Offray Vladimir Luna Cárdenas wrote: >> Hi, >> >> On the names side, I would advice some kind of unique combination, >> for example "Jupyter notebook", works fine to find particularly the >> project this community is trying to build. In my own case, I spend a >> lot of time thinking in names and I discuss them with friends with a >> drink or a meal. Despite of not having a strong web presence in the >> sense of continuous updates in the web pages of my projects (my main >> web page is still under construction!) they position relatively well >> on google and DDG engines: >> >> [1] https://encrypted.google.com/search?hl=es&q=grafoscopio (5th link) >> [2] https://encrypted.google.com/search?hl=es&q=mutabit (1st link) >> [3] https://duckduckgo.com/?q=mutabit&t=ffsb (1st link) >> [4] https://duckduckgo.com/?q=grafoscopio&t=ffsb (2nd link) >> >> It didn't happen overnight but was a fluid organic process. So may be >> "Mushroom crypto" or "Risotto crypto" could be important descriptors >> in the main web presence sites of your projects (BTW, I can't find >> the repos and GitHub answers with 404). >> >> Cheers, >> >> Offray >> >> >> On 07/12/15 17:57, Robert Withers wrote: >>> Ben, Huw, Todd and Sven, thank you all for your feedback! I suppose >>> I could call the project "CryptOCaps" but for some reason I glommed >>> onto mushroom as the name. Not grandiose and it is somewhat >>> descriptive...a network of secure sessions, each one a mushroom. >>> Ceps are highly valued. We can tag it for the catalog. >>> >>> For sure, we have Seaside, Morphic, Nebraska, Fuel, Alien, Cog, >>> Monticello and that's just the squeak side of unusual naming of >>> projects. I hope that "mushroom" gains a wide reputation as a solid, >>> reliable, secure and performant session layer under the CryptOCaps >>> presentation...I am thinking of splitting the secure session layer >>> from the ocaps presentation layer, but this would require another >>> name choice, so I hesitate...perhaps "Risotto"? What are your thoughts? >>> >>> Best, >>> Robert >>> >>> On 12/07/2015 10:38 AM, Ben Coman wrote: >>>> I like it, but it seems you missed my point :) >>>> mushroom --> 117,000,000 is two orders of magnitude more hidden. >>>> Anyway, maybe I overplay its significance. >>>> cheers -ben >>>> >>>> On Mon, Dec 7, 2015 at 11:11 PM, Robert Withers >>>> <robert.w.withers(a)gmail.com> wrote: >>>>> I renamed the project to Mushroom and I also dumped the encoding >>>>> work to >>>>> focus on shutdown, optimization and serialization. Here's the wiki: >>>>> https://github.com/SqueakCryptographySquad/Mushroom/wiki >>>>> >>>>> thanks,Robert >>>>> >>>>> >>>>> On 12/06/2015 01:42 AM, Ben Coman wrote: >>>>>> On Sun, Dec 6, 2015 at 10:42 AM, Robert Withers >>>>>> <robert.w.withers(a)gmail.com> wrote: >>>>>>> On 12/05/2015 09:24 PM, Ben Coman wrote: >>>>>>>> On Fri, Dec 4, 2015 at 11:57 PM, Robert Withers >>>>>>>> <robert.w.withers(a)gmail.com> wrote: >>>>>>>>> Now I think you are right on with your observation. >>>>>>>>> Additionally, the >>>>>>>>> number >>>>>>>>> of dialects could increase further with Fuel serialization, >>>>>>>>> just port >>>>>>>>> SecureSession and bits. >>>>>>>>> >>>>>>>>> Alright, I came up with a name and it may border on the >>>>>>>>> egregious ... >>>>>>>>> presenting ... >>>>>>>>> >>>>>>>>> "Maelstrom" >>>>>>>> Great sounding name. However some general advice for the >>>>>>>> community, >>>>>>>> since I see a lot of great sounding project names drowned out >>>>>>>> in the >>>>>>>> noise of our web-search-centric universe. A litmus test for >>>>>>>> project >>>>>>>> naming is using google search to find which return low search >>>>>>>> results. >>>>>>>> Today, its more important to be unique than any other attribute >>>>>>>> of a >>>>>>>> name. So in general, *dictionary* english words are not the best. >>>>>>>> One technique is to intentionally mispell the word you like. >>>>>>>> Here are >>>>>>>> some comparative examples (note, the surrounding quotes are >>>>>>>> required >>>>>>>> to avoid google trying to be helpful and correct the spelling)... >>>>>>>> >>>>>>>> "maelstrom" --> 7,480,000 >>>>>>>> "maelstroom" --> 6,200 >>>>>>>> "maelstrum" --> 2,280 >>>>>>>> "maelstruum" --> 7 >>>>>>>> >>>>>>>> Lots of interesting other techniques can be found by searching on: >>>>>>>> techniques to generate brand names or domain names. >>>>>>>> >>>>>>>> cheers -ben >>>>>>> >>>>>>> I would be happy to change the names to something more unique, >>>>>>> though it >>>>>>> may >>>>>>> take a few. Are you suggesting "maelstruum"? >>>>>>> >>>>>>> cheers, >>>>>>> Robert >>>>>>> >>>>>>> >>>>>> *Suggesting* yes, but the choice is yours ;) You need to own it. >>>>>> >>>>>> I think maelstruum is certainly memorable with the double "u", but >>>>>> maybe jarring next the the "m". I'm inclined to maelstroom, since I >>>>>> associate it with "zoom". I wouldn't necessarily go for the >>>>>> absolute >>>>>> lowest results. I have an entirely unsubstantiated belief that >>>>>> anything less than 10,000 gives a reasonable chance to compete >>>>>> once a >>>>>> user's browsing history is taken into account. Finally you need to >>>>>> check existing results don't return something abhorrent (I didn't do >>>>>> this). >>>>>> >>>>>> I'd encourage to play around testing on google search. Its quick and >>>>>> easy to generate and test alternatives. I've added a few more below. >>>>>> "maelstra" --> 3,560 >>>>>> "maelstram" --> 504 >>>>>> "maelstrim" --> 1200 >>>>>> "maelstroon" --> 58 >>>>>> "maelstroomi" --> 4 >>>>>> >>>>>> btw, I wouldn't swap the order of the "ae" since that would be >>>>>> susceptible to real typing errors. >>>>>> >>>>>> cheers -ben >>>>>> >>>>> >>> >>> >>> >> >> > -- . .. .. ^,^ robert

1 0

[Unicode] Summary (Re: [Pharo-dev] Unicode Support // e acute example --> decomposition in Pharo?)
by H. Hirzel 19 Dec '15

19 Dec '15

Hello Sven Thank you for your report about about your experimental, proof of concept, prototype project, that aims to improve Unicode support. Please include me in the loop. Below is is my attempt at summarizing the Unicode discussion of the last weeks. Corrections /comments / additions are welcome. Kind regards Hannes 1) There is a need for improved Unicode support implemented _within_ the image , probably as a library. 1a) This follows the example of the the Twitter CLDR library (i.e. re-implementation of ICU components for Ruby). https://github.com/twitter/twitter-cldr-rb Other languages/libraries have similar approaches - dotNet, https://msdn.microsoft.com/en-us/library/System.Globalization.CharUnicodeIn…) - Python https://docs.python.org/3/howto/unicode.html - Go http://blog.golang.org/strings - Swift, https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swif… - Perl http://blog.golang.org/strings 1b) ICU is _not_ the way to go (http://site.icu-project.org/) . This is because of security and portability reasons (Eliot Miranda) and because of the Smalltalk approach that wants to expose basic algorithms in Smalltalk code. In addition the 16bit based ICU library does not fit well with the Squeak/Pharo UTF32 model. 2) The Unicode infrastructure (21(32) bit wide Characters as immediate objects, use of UTF-32 internally, indexable strings, UTF8 for outside communication, support of code converters) is a very valuable foundation which makes algorithms more straightforward at the expense of a more memory usage. It not used to its full potential at all currently though a lot of hard work has been done. 3) The Unicode algorithms are mostly table / database driven. This means that dictionary lookup is a prominent part of the algorithms. The essential building block for this is that the Unicode character database UCD (http://www.unicode.org/ucd/) is made available _within_ the image with the full content as needed by the target languages / scripts one wants to deal with. The process of loading the UCD should be made configurable. 3a) a lot of people are interested in the Latin script (and scripts of similar complexity) only. 3b) The UCD data in XML form http://www.unicode.org/Public/8.0.0/ucdxml/ offers a download with and without the CJK characters. 4) The next step is to implement normalization (http://www.unicode.org/reports/tr15/#Norm_Forms). Glad to read that you have reached results here with the test data: http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt. 5) Pharo offers nice inspectors to view dictionaries and ordered collections (table view, drill down) which facilitates the development to table driven algorithms. The data structures and algorithm are do not depend on a particular dialect though and may be ported to Squeak or Cuis. 6) After having implemented normalization, comparison may be implemented. This needs CLDR access (collation, Unicode Common Locale Data Repository, http://cldr.unicode.org/ ). 7) An architecture has the following subsystems 7a) Basic character handling (21(32)bit characters in indexable strings, point 2) 7b) Runtime access to the Unicode Character Database (point 3) 7c) Converters 7d) Normalization (point 4) 7e) CLDR access (point 6) 8) The implementation should be driven by the current needs. An attainable next goal is to release 8a) a StringBuilder utility class for easier construction of test strings i.e. instead of > normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 > 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: > String). do normalizer composeString: (StringBuilder construct: 'Du\u0308sseldorf Ko\u0308nigsallee') and construct some test cases with it which illustrate some basic Unicode issues. 8b) identity testing for major languages (e.g. French, German, Spanish) and scripts of similar complexity. I 8c) to provide some more documentation of past and concurrent efforts. Note: This summary has only covered string manipulation, not rendering on the screen which is a different issue. On 12/16/15, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > Hi Hannes, > > My detailed comments/answers below, after quoting 2 of your emails: > >> On 10 Dec 2015, at 22:17, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >> >> Hello Sven >> >> On 12/9/15, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >>> The simplest example in a common language is (the French letter é) is >>> >>> LATIN SMALL LETTER E WITH ACUTE [U+00E9] >>> >>> which can also be written as >>> >>> LATIN SMALL LETTER E [U+0065] followed by COMBINING ACUTE ACCENT >>> [U+0301] >>> >>> The former being a composed normal form, the latter a decomposed normal >>> form. (And yes, it is even much more complicated than that, it goes on >>> for >>> 1000s of pages). >>> >>> In the above example, the concept of character/string is indeed fuzzy. >>> >>> HTH, >>> >>> Sven >> >> Thanks for this example. I have created a wiki page with it >> >> I wonder what the Pharo equivalent is of the following Squeak expression >> >> $é asString asDecomposedUnicode >> >> Regards >> >> Hannes > > You also wrote: > >> The text below shows how to deal with the Unicode e acute example >> brought up by Sven in terms of comparing strings. Currently Pharo and >> Cuis do not do Normalization of strings. Limited support is in Squeak. >> It will be shown how NFD normalization may be implemented. >> >> >> Swift programming language >> ----------------------------------------- >> >> How does the Swift programming language [1] deal with Unicode strings? >> >> // "Voulez-vous un café?" using LATIN SMALL LETTER E WITH ACUTE >> let eAcuteQuestion = "Voulez-vous un caf\u{E9}?" >> >> // "Voulez-vous un café?" using LATIN SMALL LETTER E and >> COMBINING ACUTE ACCENT >> let combinedEAcuteQuestion = "Voulez-vous un caf\u{65}\u{301}?" >> >> if eAcuteQuestion == combinedEAcuteQuestion { >> print("These two strings are considered equal") >> } >> // prints "These two strings are considered equal" >> >> The equality operator uses the NFD (Normalization Form Decomposed)[2] >> form for the comparison appyling a method >> #decomposedStringWithCanonicalMapping[3] >> >> >> Squeak / Pharo >> ----------------------- >> >> Comparison without NFD [3] >> >> >> "Voulez-vous un café?" >> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. >> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter >> asString, '?'. >> >> >> eAcuteQuestion = combinedEAcuteQuestion >> false >> >> eAcuteQuestion == combinedEAcuteQuestion >> false >> >> The result is false. A Unicode conformant application however should >> return *true*. >> >> Reason for this is that Squeak / Pharo strings are not put into NFD >> before testing for equality = >> >> >> Squeak Unicode strings may be tested for Unicode conformant equality >> by converting them to NFD before testing. >> >> >> >> Squeak using NFD >> >> asDecomposedUnicode[4] transforms a string into NFD for cases where a >> Unicode code point if decomposed, is decomposed only to two code >> points [5]. This is so because when initializing [6] the Unicode >> Character Database in Squeak this is a limitation imposed by the code >> which reads UnicodeData.txt [7][8]. This is not a necessary >> limitation. The code may be rewritten at the price of a more complex >> implementation of #asDecomposedUnicode. >> >> "Voulez-vous un café?" >> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. >> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter >> asString, '?'. >> >> >> eAcuteQuestion asDecomposedUnicode = >> combinedEAcuteQuestion asDecomposedUnicode >> >> true >> >> >> >> Conclusion >> ------------------ >> >> Implementing a method like #decomposedStringWithCanonicalMapping >> (swift) which puts a string into NFD (Normalization Form D) is an >> important building block towards better Unicode compliance. A Squeak >> proposal is given by [4]. It needs to be reviewed.extended. >> >> It should probably be extended for cases where there are more than >> two code points in the decomposed form (3 or more?) >> >> The implementing of NFD comparison gives us an equality test for a >> comparatively small effort for simple cases covering a large number of >> use cases (Languages using the Latin script). >> >> The algorithm is table driven by the UCD [8]. From this follows an >> simple but important fact for conformant implementations need runtime >> access to information from the Unicode Character Database [UCD][9]. >> >> >> [1] >> https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swif… >> [2] http://www.unicode.org/glossary/#normalization_form_d >> [3] >> https://developer.apple.com/library/ios/documentation/Cocoa/Reference/Found… >> [4] String asDecomposedUnicode http://wiki.squeak.org/squeak/6250 >> [5] http://www.unicode.org/glossary/#code_point >> [6] Unicode initialize http://wiki.squeak.org/squeak/6248 >> [7] http://www.unicode.org/Public/UNIDATA/UnicodeData.txt >> [8] Unicode Character Database documentation >> http://unicode.org/reports/tr44/ >> [9] http://www.unicode.org/reports/tr23/ > > > Today, we have a Unicode and CombinedCharacter class in Pharo, and there is > different but similar Unicode code in Squeak. These are too simple (even > though they might work, partially). > > The scope of the original threads is way too wide: a new string type, > normalisation, collation, being cross dialect, mixing all kinds of character > and encoding definitions. All interesting, but not much will come out of it. > But the point that we cannot leave proper text string handling to an outside > library is indeed key. > > That is why a couple of people in the Pharo community (myself included) > started an experimental, proof of concept, prototype project, that aims to > improve Unicode support. We will announce it to a wider public when we feel > we have something to show for. The goal is in the first place to understand > and implement the fundamental algorithms, starting with the 4 forms of > Normalisation. But we're working on collation/sorting too. > > This work is of course being done for/in Pharo, using some of the facilities > only available there. It probably won't be difficult to port, but we can't > be bothered with probability right now. > > What we started with is loading UCD data and making it available as a nice > objects (30.000 of them). > > So now you can do things like > > $é unicodeCharacterData. > > => "U+00E9 LATIN SMALL LETTER E WITH ACUTE (LATIN SMALL LETTER E ACUTE)" > > $é unicodeCharacterData uppercase asCharacter. > > => "$É" > > $é unicodeCharacterData decompositionMapping. > > => "#(101 769)" > > There is also a cool GT Inspector view: > > > > Next we started implementing a normaliser. It was rather easy to get support > for simpler languages going. The next code snippets use explicit code > arrays, because copying decomposed diacritics to my mail client does not > work (they get automatically composed), in a Pharo Workspace this does work > nicely with plain strings. The higher numbers are the diacritics. > > (normalizer decomposeString: 'les élèves Français') collect: #codePoint as: > Array. > > => "#(108 101 115 32 101 769 108 101 768 118 101 115 32 70 114 97 110 99 > 807 97 105 115)" > > (normalizer decomposeString: 'Düsseldorf Königsallee') collect: #codePoint > as: Array. > > => "#(68 117 776 115 115 101 108 100 111 114 102 32 75 111 776 110 105 103 > 115 97 108 108 101 101)" > > normalizer composeString: (#(108 101 115 32 101 769 108 101 768 118 101 115 > 32 70 114 97 110 99 807 97 105 115) collect: #asCharacter as: String). > > => "'les élèves Français'" > > normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 > 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: > String). > > => "'Düsseldorf Königsallee'" > > However, the real algorithm following the official specification (and other > elements of Unicode that interact with it) is way more complicated (think > about all those special languages/scripts out there). We're focused on > understanding/implementing that now. > > Next, unit tests were added (of course). As well as a test that uses > http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt to run about > 75.000 individual test cases to check conformance to the official Unicode > Normalization specification. > > Right now (with super cool hangul / jamo code by Henrik), we hit the > following stats: > > #testNFC 16998/18593 (91.42%) > #testNFD 16797/18593 (90.34%) > #testNFKC 13321/18593 (71.65%) > #testNFKD 16564/18593 (89.09%) > > Way better than the naive implementations, but not yet there. > > We are also experimenting and thinking a lot about how to best implement all > this, trying out different models/ideas/apis/representations. > > It will move slowly, but you will hear from us again in the coming > weeks/months. > > Sven > > PS: Pharo developers with a good understanding of this subject area that > want to help, let me know and we'll put you in the loop. Hacking and > specification reading are required ;-) > >

3 2

← Newer
1
2
3
4
5
6
7
8
9
...
17
Older →

Jump to page: