Or: Where things went wrong, and where things went right.
Parrot's not dead or anything, so far as I know. There's too much riding on it for too many people, and last I checked (which, granted, was months ago) it was still going, so this isn't really a post-mortem on the project, but rather on my tenure with it. (Which is dead) This, I'm sure, isn't all of it, but I've been jotting things down as the time's passed and filled in the explanations for the bits and pieces. Hopefully it'll help other suckersvolunteers who might run projects in the future.
Trusting the code gnomes: Good
This is something that worked out pretty well -- that is, checking in code that was barely OK, or minimally functional, and trusting that people would take the skeleton and flesh it out. This happened a lot, and it was quite nice to see.
It's not at all surprising. A lot of what we were doing in parrot was scary for a lot of people. Unnecessarily scary, but that's irrelevant -- we were doing Bizarre Dark Magic, and it didn't matter what we said, actually creating some of that from nothing was more than almost anyone thought they could do. On the other hand, it's far less scary to take a piece of working Dark Magic and change it around some, patch it up, and make it do more. That was relatively easy for a lot of folks.
It took me a while to recognize this, but once I did it was helpful, both in getting stuff done and in getting new people involved. There was a pool of "well, that stuff kinda works, but it's nasty and if it did X too it'd be great" code folks could poke at, as well as a good chunk of code that got checked in with "barely functional but works" checkin messages. Those tended to get quickly rewritten by folks watching the list that wanted to help and saw something they could comfortably work on.
Keeping an eye on the prize: Good
This was a constant fight, made worse since a lot of people didn't do this. (And in part made worse because the long-term plans were detailed in a dozen or more different spots making it difficult for anyone to keep track) Damn near everything I designed in parrot was done with an eye towards what I wanted the total package to look like. That meant juggling threads, Unicode, alternate character encodings, asynchronous IO, potentially asynchronous garbage collection, continuations, events, notifications, perl's dynamic data, all the new stuff Larry was pondering for Perl 6, along with all the stuff perl 5, python, and ruby do now.
This is one of those things that's tough for most people to manage, not because they're not capable (dunno if they are or not. I assume people are) but because to do it means having the whole design in your head, and that's tough if you're not the person doing the design. It's doubly tough if some of the design goals are things you either just don't care about, or actively dislike, and we had a lot of both going on with various people.
Skimping on the docs: Bad
This was a pretty constant refrain -- where are the docs? Well, I didn't write enough of them, and the ones I did write tended to spark fights, either because people strongly disagreed or they weren't clear enough. Unfortunately I found it tough to grab the time I really needed to get the documents written, which was a constant problem.
It didn't help that people tended to ignore the documents once they were written, which pissed me off more than once. Well over half the complaints about docs being out of date were because the code as implemented didn't match things as specified in the docs and the docs were what was supposed to be happening. (This is important -- don't whine about out of date docs or missing docs if you've gone out of your way to ignore the specifications as written)
Not getting an HLL up and running and maintaining it: Bad
Assembly language is fine as far as it goes, and having an assembler done Real Quick(tm) was damned important. What I should have done shortly after that was to get an officially blessed and maintained Parrot Twiddling Language that would be usable by anyone who isn't keen on really low-level stuff. Parrot got a lot of Perl people (and later some Ruby and Python people) but Parrot itself was written in C, and a lot of the low-level hackery needed to be done in C.
There was, though, a massive amount of stuff that didn't need to be done in C. If we'd had a high-level language ready to go nearer to the beginning then we'd have had a much better handle on the compiler interface, a place for people to play much sooner, an easier way to write tests, and a good leg up on the standard library.
The gee-shucks architect: Good and bad
One of the things I decided early on was to downplay my own skills.
Let's be blunt and up front here. I'm good at what I do, and I do a lot of things that a lot of people can't manage. Parrot's first mark and sweep garbage collector was put together in about four hours while I was sitting in my local public library (no wifi but comfy chairs) and it worked pretty well. The hardest part about it was doing the dull stuff involved with it. I'm reasonably certain that's not normal. And no, I don't know how to say "Yeah, I'm good at doing that stuff that makes your head explode -- it's easy" without coming across as an arrogant prat, so I don't. (Say it. I've no idea about the arrogant prat bit)
On the other hand, I'm usually really uncomfortable being up front about what I'm good at (much to a number of people's deep annoyance), and I really, really didn't want people afraid to touch code I'd written because they thought it required some sort of skill at brain surgery to alter.
One of the running jokes was that we'd know Parrot was ready for release 1.0 because all the code I'd written and checked in had been rewritten. I cultivated this, to the point of occasionally checking in bad (working, but crappily written) code on purpose, partially to give the code gnomes something to gnaw on and partly to enhance my rep as an adequate but not great coder. If people thought the code was deeply magic they wouldn't want to touch, and I knew we needed people not afraid to touch.
This was partly a good thing. We got a lot of people who hacked in on parts of an interpreter system that they might not have otherwise dared touch -- if I could deal with it and I was just OK as coder then they could dig in. We got a number of people who might not otherwise get involved actually get involved because of this, and that was a good thing.
Unfortunately the downside there is that I lost a lot of respect from some people, because I was viewed as, at best, an adequate coder. When you've got people who's sole measure of personal worth is the code they produce, well... you can see the problem.
Overcommitting the architect: Bad
I had a full-time job (with more than full-time time commitments), a family, a marriage that's been rocky for ages, and Parrot on my plate of things to do. Any one of those things was enough by itself, and two would keep anyone busy. I had all four, and they all suffered in one form or another. (Parrot, unfortunately, didn't always get the short end of the stick either) Bluntly I had more things to do than any one person could reasonably do, and I didn't have the sense to back out of some of my commitments until things got past bad for me.
Running a project like Parrot, where the scale's damn big, requires a minimum amount of attention, and for a while I didn't have that attention to give it, and what attention I did have was mostly wasted fighting with Leo.
Professional courtesy: Good
One of the things I insisted on was that people behave professionally. (Not, mind, as adults -- I know plenty of adults who behave badly, and some of the kids (and yes, Brent, you and Zach counted at the time :P) involved with parrot behaved better than some of the older adults) I wasn't looking for the sort of cult of personality that pervades the perl, python, and ruby development camps -- I find that rather distasteful, both as the potential target and as a general participant. I also remembered the perl5-porters mailing list when it was more a shark tank than a development mailing list, and I did not want that.
All-volunteer or not, I viewed parrot as a professional engineering project, and I wanted people to behave accordingly. For the most part everyone did, and I was happy about that.
Not telling people to shut the fsck up and cope: Bad
Professional courtesy is fine and all, but if you're running a project, with volunteer labor or not, sometimes you need to tell people to suck it up and deal. More importantly, there's nothing wrong with a judicious exercise of authority, especially if the exercise of authority is part of the job you hold. I was the architect for the project and it was my responsibility to design the system and get that design implemented. Part of that meant that yes, I should have told people to cope. Shying away from confrontation doesn't do anyone any good -- I did, and the project suffered for it.
People not shutting up and dealing: Bad
This one was, to some extent, out of my control, but it was still a problem, especially with Leo. If you're going to take part in a project, that's fine. If a feature or something is under development and you want to chip in, that's fine too. When the decision is made, though, shut up and cope if you disagree. If it's not your call, then once the call's made you deal with it. Odds are the person who made it also has some issues with it, but there are reasons for it to have been made, reasons you may be unaware of, or not understand. Regardless, projects don't go anywhere if decisions keep getting rehashed over and over again.
This was in part because of me not putting my foot down enough, but even in the cases where I did it was often ignored, and that was a problem. If you don't like the decisions being made in a project you register your objections when appropriate, and if things don't work the way you think they should you go away and find something else to do. The world's a big place with a lot of things going on, and nobody in a volunteer project needs to deal with you bitching about or subverting decisions you don't like. Register your complaint and either cope or go away.
Deferring to contributors: Good and bad
Parrot was all-volunteer, and because of that I didn't push people much, and took what I could get. It made sense, since how could I reasonably put any pressure on people who were donating their time and efforts? Well, I should have, at least more than I did.
When you're running a volunteer project, there's nothing wrong with asking people to do things, and expecting that they'll do them if they say they will. People are sensible and know what they can do, and will either step up or not. While you can't demand people do something they've not volunteered for, if they say they will, then you have every right to expect they will do it, and it's fine to ask people to do something as long as you can take no for an answer. Being volunteer does mean people may bail on you with little or no notice, and it means that you take second (or third) place behind other activities, so you need to keep on top of who's doing what, but there's nothing wrong with asking.
I did my best to respect people's other commitments -- everyone's got a life outside Parrot and honestly if the choice was someone doing parrot and making a mess of their home life or bailing on us, I'd be happy to forcibly kick people out to go deal with the important stuff. Not that I had to, but Parrot was just software and on the whole software's just not that important, not in the grand scheme of things.
Keeping mum about Perl 6: Bad
Mmmm, perl 6, the original reason Parrot started, and after a couple of years a nearly irrelevant thing in parrot development. I hear it's more important now, which is fine.
I'll be blunt. I don't give a damn about perl 6 at this point. Haven't for years. I'm not a big OO guy (when we started I had a knee-jerk dislike of objects, a problem I've since shed, though as a performance guy I think they're overused) and perl 6 was getting deeply OO during the design. Plus dealing with Larry as a designer was... well, it was a pain in the ass. I finally gave up in disgust on perl 6 when we lost over a week of Larry's thinking about perl time to the 5 disc DVD releases of one of the Lord of the Rings movies. (The first one, I think, though it's been years)
There was also a lot of creative tension at times in the whole design process, and I hold the dubious honor of being the only person I know of, outside his kids, to get Larry mad. (And yes, he does get mad, though in a nice way. Go figure) This is one of those "laws and sausages" things for most people. Does it matter to you how much waffling Larry's done, or how many times (and over what) he and I just-barely-didn't-shout at each other, or how many things I flat-out told him I wasn't going to implement if he designed them in, or implement even if he forbade it, or how bloody long some things took? No, it really doesn't. What ultimately matters is the final result, the rest is just development crap, and no different than you see in a lot of other projects.
Not, mind, that I think Perl 6 is going to be bad. I don't. I respect Larry immensely as a designer -- he's good, and on some days he hits great, and I say that even disagreeing with some of his decisions. There are a number of good people working to make the design happen, too.
I just don't give a damn, but I kept mostly silent about it, and I think that, for me, that was a mistake. Others may disagree, which is fine, but the perl 6 development process has not been trouble-free, and I think it could stand to have a lot more light shined on it. Looks like that's happening more now, which is a good thing. It just should've happened earlier.
Leo: Bad
Normally I wouldn't name names -- it's unprofessional and a bit unbecoming. Unfortunately you'd have to be completely uninvolved with parrot or desperately clueless to not know how well we did, or rather didn't, get along. Leo was the single biggest mistake I made with the project.
Leo, bluntly, is a massive pain in the ass, and because of him Parrot was about a year behind where it could've been when I left. I spent far more of what energy I had dealing with him than anything else, rehashing old settled design decisions over and over again, putting up with snide comments, shots on the design (often complaining about missing design documents that actually existed), and whines about the way things were designed and how badly they were. Something, bluntly, I found infuriating since most of the things he complained about were for languages that he didn't program in. This would include perl, python, and ruby. Leo, as someone who never used any of those languages, knew better than those of us who've worked on the cores of one or more of them what was good and bad.
Let's be clear. Yes, Leo writes a lot of code. Yes, he goes and implements features. Those are good things under normal circumstances. Unfortunately his code's difficult to get into, not all that great, and puts off anyone who wants to modify it, pretty much leaving any system he's worked on impenetrable to most anyone else. The features he does implement fresh didn't follow the documentation for those features, and he would re-implement the same system over and over again rather than working on something new. His interpersonal skills drove a lot of people off the project as well -- I'm far from the first who left Parrot because of Leo. Bluntly, Leo was far more of a hindrance than a help, and I put him in the position to kick the crap out of Parrot, so I've nobody to blame but myself here.
Yes, I know, volunteer project, and the third (or is it fourth) rewrite he did on the garbage collection system's fast! Woo! But... so what? In the mean time dealing with him wasted so much time and energy that exceptions, IO, and events never got dealt with, a dozen or more good people went away permanently, a large chunk of parrot's a bloody mess, and a lot of it is prematurely optimized into near-unfixability.
I shouldn't have made Leo pumpking when I knew he'd driven off other developers, and I should've told him to go away the first time I ignored my perl6-internals mail for a week because I couldn't deal with him. I didn't, and Parrot suffered.
The wrapup
So, what did I learn from all this?
I expect there's more, but there you go, and take it for what you will.
'Tis the season, around here, to pick apples. Yeah, you can go to the market and get them pretty much year 'round, but there's something nice about being able to go to one of the local orchards and grab a bushel of local apples. Besides supporting the local farmers, you can get a lot of heirloom or minor varieties of apples this way, varieties that're really tasty but don't necessarily transport or store well enough to be commercially distributed.
I'm a big apple fan -- apple pie, apple turnovers, applesauce, apple ice cream, apple fritters (yum!), apple jelly -- it's all good. One thing I've been looking for is something suitable for throwing over ice cream, something like hot fudge or caramel sauce, only apple. (This would be an apple sauce, rather than applesauce. The former has tasty goo with chunks of apple in it, the latter is mashed up apples)
My first try didn't quite work out. Almost, but not quite, and the kids really liked it, so I won't call it a failure, rather differently successful.
Thick Apple Glop
1 pound apples (probably 3) cored, peeled, and diced
2T butter
1/4 c brown sugar
1/4c water
1t cinnamon
1/2t nutmeg
Melt butter over medium heat in a large skillet until it starts bubbling. Toss in diced apples and fry until they're soft. Toss in sugar, cinnamon, water, and nutmeg. Stir and cook another two or three minutes, until all the ingredients are incorporated, most of the water's driven off, and it's all nice and bubbly. The longer you cook this the more the sugar will caramelize and the apples will soften, so you've got some leeway here. (Don't burn the sugar -- it tastes nasty and will ruin the thing) You're not going to get a good caramel here, but that's OK.
Now, I'd hoped that between the melted sugar, butter, and liquid from the apples that I'd make up a nice sauce. No joy -- the result, when it cooled, was darned thick. Really tasty, but thick. Not what I was looking for, though it was really good.
I thought I'd see about reducing some apple cider (that is, simmering it until some of the water's driven off, leaving a thicker liquid behind) and throwing that in, but apple cider doesn't reduce. Instead the solids precipitate out leaving something that looks like brown miso soup. (And is, itself, quite tasty, but still really thin) No joy there.
Still, the glop tasted really good, it just needed some sauce around it. The reduction of apple cider didn't work by itself, but apples have a lot of pectin in them. Pectin, like gelatin, makes things thicker, so I figured that if I added more liquid into the glop, I'd have better luck, and I did, and boy is it good:
Thick Apple Sauce
1 pound apples (probably 3) cored, peeled, and diced
2T butter
1/4 c white sugar
2c apple cider
1t cinnamon
1/2t nutmeg
As before, melt the butter, fry the apples, and throw in everything else. Then let the stuff simmer until you lose about half the liquid. If you've never done this before, that means you use a medium heat, get the liquid simmering (that is, lots of little bubbles, not a huge roiling boil. A boil will burn things and turn 'em nasty) and leave it alone! That's the hard part, the not touching. You want to stir it every minute or so, but that's about it. When you've lost about half the liquid you're done. The sauce will still be thin, but that's fine -- pectin thickened sauces get more solid the colder they get, so as the sauce cools it will thicken. (If you've only done flour or corn starch thickened sauces this will be a little odd to you, but that's fine. Consider it a learning experience) This version uses white sugar instead of brown sugar, since there was a bit of a brown sugar taste to the glop and I wanted a cleaner apple flavor.
Note that a pound of apples gives off a lot of pectin, so if you want a thinner sauce, or more sauce, either add in more cider (I think the recipe can manage up to a quart of cider, but I've not tested that) or don't let it reduce as long. It still will need some time to pull the pectin out of the apples, so you can't just cook the apples, throw the rest of the stuff in, and decant it. You must simmer it at least a few minutes.
Do please note that both nutmeg and cinnamon, as they're powdered tree bits, take some time to fully hydrate and release their flavor. Fresh out of the pan the mix will feel a little gritty and taste OK. After it sits (preferably overnight) the grit will be gone and the flavor mixing in nicely. Mmmm!
No, there's no computer content here. Deal with it, this stuff's tasty. :-P
I've been working on $CONSULTING_GIG (ex-$WORK_GIG)'s compiler today, digging into a performance issue that it's got. It doesn't show in too many places -- most of the forms are really snappy -- but when it does... ouch.
The problem, in this case, is one of constants.
Now, constants are one of those facts of life, and that's fine. I rather like them, and when given a bit of a chance there's all sorts of interesting things one can do with a program in a compiler. Other than some simple constant folding, I didn't do any of them, mostly for time reasons.
The way I implemented constants was pretty straightforward, since that's easiest to debug. Because all the binary operations had to be overloaded, and I didn't want an explosion of overload functions, the compiler generates PMCs of the appropriate language type for the constants, something that is certainly quite common -- just because the source says:
a + 4
you don't necessarily want a low-level integer for that 4, since you may have to allow people to overload operations in the integer class and have that affect integer constants. (You can do this in ruby, for example) This is no big deal.
What I'd done is taught the compiler to emit the appropriate code to create a new PMC and give it a value before each use, so the above statement would read in PIR something like:
I16 = find_type "Int32"
P16 = new I16
P16 = 4
P17 = global 'a'
P18 = new I16
P18 = P16 + P17
and go on from there. No big deal. Except... my victim program of the moment was taking something like two seconds a line item to pull up and display data, which was nasty. Yeah, sure, that was accessing a Postgres database hosted on a windows box over an 802.11b wireless link, but still, nasty. (It was 5 seconds a line item before I optimized the generated SQL -- Postgres has a hard time with "field LIKE $1" and index usage, since the query optimizer gets the query to optimize but not the value of the placeholder variables, so things get messy, but that's a separate issue)
Since nothing big and obvious was taking all the time, it was time to start in on the hopeful stuff and see where we went. In this case, constant creation seemed an obvious spot to look. Besides the cost of creating all the new PMCs every time, there's the disposal cost to allow for as well, since all those temp values need to be cleaned up by the garbage collector. Which it does, and reasonably well, but it's a lot faster to not do something than to do it.
The worry was that creation would be cheap enough that it was less than the cost of looking the variables up in the global namespace, which is a definite worry as parrot's hashing code has been sub-swell in the past. Still, it only took a day to make the changes and test everything out to make sure it worked.
Net result? Reusing constants rather than recreating them cut the time the victim program took to load up its line items from 17 seconds down to 11. Not bad, all things considered, and definitely surprising. I'd expected to shave off a second or so, maybe, which still would've been worth the effort, but not 6. That was nice.
So, net thing learned? Pre-generating constants is often worth it. (Now I'd love to find out what I could shave off the run time by changing all those by-name lookups to by-offset lookups...)