Programming

A short while ago, I posted an article here arguing that model-to-model transformations (or rather, modularity features based on them) are necessary to escape a new "tyranny of the dominant decomposition" in model-driven programming. Well, what should arrive in my electronic mailbox a few days ago but a newsletter from MetaCase wherein the first big headline reads, provocatively, "Model-To-Model Transformations: A Recipe for Trouble." Oh my!

The newsletter links to an article posted a couple of months ago by Steve Kelley on the Code Generation Network site. In a nutshell, Kelley argues "[o]ur experience is that it is better that generators produce code directly, rather than producing intermediate models that you need to extend during the development process." He has a lot more to say than this, of course, and the whole editorial is definitely worth your time.

The Whole Article

He spends some time laying out the benefits of code generation for product line development (though he distinguishes this from a one-off project, where "the payback may be little greater than the effort required to automate"). And he explains why a UML-based approach, or one based on any "general-purpose design language" for that matter, is doomed to the same fate as those products of yesteryear that have "taught many developers to look at code generators with suspicion." He argues that only domain-specific languages created by a product line's architects themselves can appropriately model features of the domain and serve as a basis for handcraft-quality code generation:

If we wish to raise the abstraction level beyond code, yet still be able to generate fully working code, going domain-specific is the only alternative.... Despite the existence of multiple ways to write code for a certain behavior, vendors usually choose just one of them, which is unlikely the ideal candidate for your specific contingency.... Third-party generators often don't know enough about an organization's specific requirements to generate ideal code, so it is not surprising that many have found generated code unsatisfactory.

Kelley then advocates simplicity as a central principle in the development of code generators based on domain-specific models. "The simpler they are to start with, the simpler it will be to make changes to them later." He discusses ways of keeping generators simple (depend on model validation rather than validating input, isolate repeating code patterns into libraries), and he prescribes the use of "a good reference implementation" as a starting point for implementing any model-to-code transformation. All of this is quite useful and appropriate advice for those using the current generation of tools, and there are many people exploring or working in this blossoming field who would doubtless benefit greatly from it.

Night and Day

As it turns out, though, our seemingly conflicting points of view amount to apples and oranges. Because when Kelley talks about model-to-model transformations, he's talking about a different way of automating software development than I am. He's talking about a user-modified, bidirectional train wreck that we both find repugnant, and for the same reasons:

Normally, the idea of model-to-model transformation is that each piece of data in the high-level language gets transformed to more than one piece of data in the lower-level language (let's say 2 pieces).

We've already parted ways here, by the way, as this passage describes the primary purpose of model-to-model transformations as the generation of lower-level models; a strictly downward-directed process. The kinds of technologies I highlight in my post are concerned with transformations in any direction, and most of the examples are horizontal, not vertical. I want a modularity strategy for transformations to be just as general as the modularity I've enjoyed in OOP languages, which have allowed me to gracefully model relationships that are horizontal, vertical, and everything "in between."

This is fine if you never (or rarely) look at the low-level language, and never (or very rarely) edit it. But if you decide to edit it, you are then working with two pieces of data, but clearly they are not totally independent, since they could be produced from one piece. The result of this is that any changes to the higher level would need to map those changes back to lower levels too.

Often people say next "and we want to be able to change the high-level models still, and have those changes reflected in the low level models". That means you are working with 1+2 pieces of information, plus that you need to come up with some way to propagate the changes down to the low-level models correctly. "Correctly" here means: without destroying information manually added to those low-level models; updating the automatically generated parts; creating new generated parts; updating manually added parts to reflect changes in the top-level models (e.g. if the manually added part refers to the name of an element originally from the high level model, and that element's name has now been changed).

Horrors, indeed! A setup like this violates a critical characteristic of successful generative methods: the inviolate nature of generated artifacts. Kelley briefly mentions an example of this characteristic elsewhere in his article, saying, "A DSM generator['s] output will not normally be edited – or even checked – by hand." So, of course, he finds such approaches untenable, as do I.

If this reflects, as Kelley suggests, the current state of the art in model-to-model transformations, I'd be sad but not surprised. But I'm also hopeful and eager for the long-term potential presented by some of the technologies currently being nurtured academically.

And lest you thought Mr. Kelley's customers stopped there, there's more...

And next, people say "and we want to be able to change the low-level models, and have the high-level models update". This is even harder.

I'LL SAY IT IS! Now my earlier sadness has shifted focus from the model business to its customers.

Conclusion

Obviously, Kelley and I are talking about different things. He's looking at current practice, particularly people who think they want "model-to-model transformations," and he's rightly turning up his nose. What I'm interested in, on the other hand, are ways of modularizing this new abstraction layer bestowed upon us by generative and model-driven programming, so as to achieve leverage in ways analogous to those with which I've been successful using currently popular programming methods. I think the best way to do this in the new world is the same as the best way in the old: separation of concerns. And I think those concerns have to be realized in a way that cross-cuts implementations. The kinds of designs that can accomplish this tend to use model-to-model transformations, but they don't use them like code generators. Rather, they generate "intermediate build products" (in terms of current tooling nomenclature) that are inherently inviolate and of interest only to the development software itself, not to humans.

The only other way in which Kelley's post can be said to clash with mine is in his emphasis on "[k]eep[ing] it simple" when it comes to transformations in general. He and I agree that small, domain-specific models yield the best results. But because of his (and our) bad experiences with CASE technologies and their putrid generated code, he emphasizes the simulation of handcraft as a dominant theme in the design of transformations, proclaiming that ideal generated code "gives a good impression, as it looks familiar, follows the required programming mode, includes appropriate comments and follows the local standards for coding style." And this emphasis leads straight to simple, reference-implementation-based transformations. Naturally!

But I'm focused on the wider, future potential offered by this new abstraction layer, and the types of modularization of models I've already found that I need, and I see way more powerful ways to automate than the methods undertaken by the people Kelley is talking to. I'm even willing to sacrifice a little quality in the code that's ultimately generated, at least in the short term, to get there. Some may argue that more than a little quality loss must inherently follow from such flights of abstraction, but I disagree. Sure, we went through an era of Visual BASIC p-code before arriving at JITs, but now everything's copacetic. The problems that need to be solved may be more abstract than those that came before, but I think they're tractable, and I look forward to seeing how people do it.

Among the papers related to generative and model-driven programming I've read over the last couple of years, one old favorite is The Side Transformation Pattern - making transforms modular and re-usable by Edward Willink (the FOG guy) and Phillip Harris. The full text is no longer available for free on-line, unfortunately, so I'll attempt to summarize, quoting the authors:

Transformation is often presented as a complete activity. This impression is particularly prevalent in today's "MDA-compliant" tools, which perform a one-stage rewrite of source ... concepts. Such an approach ... offers little opportunity for evolution through progressive transformation.

Where a single stage approach satisfies the user's requirements, it can offer dramatic improvements in productivity, for example in code generator templates that produce Java or C++ source text directly from UML graphical models. Too often however, the one-stage approach inhibits a proper separation of different programming concerns. The result is a lack of flexibility and visibility in the transformation process, leading to poor opportunities for enhancement or re-use.

They may be talking about MDA, but what they're saying applies equally to almost every tool in this field and most of the projects in development. The commercial tools, in particular, seem to concentrate most of their design efforts on graphical model editors, and they treat the actual generation of artifacts as a simple, one-stage process, usually taking the form of text template languages. If you're lucky, they may incorporate some other rudimentary features, like model validation and repositories.

This insight into the inadequacy of current approaches is valuable enough on its own, but the really beautiful thing about this paper is that it's a pattern paper. It presents a solution (or, at least, a part of a solution) in the form of an easily understood, straightforwardly replicated design pattern. How many pattern papers do you see for implementors of these kinds of tools? This is the only one I'm aware of.

We therefore look to an approach that involves many small transformations, each dealing with a single aspect of the problem, such that relevant transformations extracted from a large library of re-usable transformations can be exploited to build an apparently custom composite with few truly custom contributions.

They're talking about leverage in the implementation of the application development tools themselves, here (the "factory," in Microsoft's lingo), not just the applications they produce. And they're achieving it through modularity and aspectual decomposition.

The authors go on to propose a pattern for the modularization of transforms involving forking and merging models, and they demonstrate the pattern in a few credible examples, including the addition of type inference constraints to a model and the insertion of synchronization barriers in a flow graph. The examples are concrete. The benefit they convey is tangible.

Are these guys (and some other folks I'll get to) onto something? Is "separation of different concerns[,] ... flexibility and visibility" valuable enough to invest so much in transformation? More after the fold.