Butler in Chicago

I'm a bit late in getting this out, but such is life when moving and attending a wedding. :-)

We had a good Core Conversation on Butler in Chicago. Sadly Dries ended up not being able to attend, but just about everyone else was on board with the roadmap. The roadmap itself hasn't changed much since Copenhagen, but things are moving again. There was also considerable talk of very, very interesting tie-ins to the configuration/deployment core initiatives, as having a more robust pre-database configuration option should help resolve a lot of the challenges we were looking at.

I'll attach the slides from my presentation to this post when I get home tonight.

Next steps:

Context

There is a partially completed context system in the Butler project on Drupal.org. Right now the handler registration system and response system are in place, but there are still some gaps and bugs. I will fill out some issues in the issue queue shortly. If you want to hep with the Butler project, this is an excellent place to jump in. Just remember to read through the architecture discussions in this group first. :-)

On a partial tangent, David Naughton (nihiliad) and I have been talking about leveraging the PECL HTTP module for interfacing with the HTTP request. That of course introduces a dependency on PECL... unless we have a user-space implementation of it. He's working on that now as a stand-alone project.

The idea is that we could include a user-space implementation of a known and tested PHP module that other developers are already familiar with in Drupal core, and use that. But, because it's OO, we can lazy-load it. If you install the PECL module, though, suddenly that part of your code becomes C, not PHP, which is faster. It probably won't have a huge impact here, but it's a trial balloon to see how viable this approach is for other PECL modules. Stay tuned. :-)

Plugins

OMG everyone seems to want this, even moreso than the unified context system. I still believe that we need the context system first and then we build plugins on top of them.

However, one of the things we discussed is not yet-again reinventing the wheel. In particular, while I have code partially working and we have the ctools model available to us, it is worth our time to try and investigate other systems to see how well and how cleanly they address the problems we need to solve.

Fortunately, a number of people stepped forward to help with that research. Specifically:

• Sam Boyer and Dmitri Gaskin are researching Symfony 2.
• Marcus Deglos (Manarth) is researching Kohana.
• Justin Randall (bejeebus) is researching Zend Framework.

And of course we should be looking at ctools as well. Mind you, I don't foresee us directly using any of the above systems per se. We're not introducing a dependency on Kohana. :-) However, we do want to try and learn from other systems so that we can build a more robust, thought-out system ourselves. Researchers, please post your results in this thread.

So the question is, what exactly do we want to research? I see our plugin system needing the following from a 10,000 foot perspective:

1. Definition of a type of plugin (eg, Cache)
2. Definition of a plugin that supports that type (eg, Memcache)
3. A way to define and store configuration for a particular instance of a plugin (eg, Memcache + the bin it is attached to).
4. A way to retrieve a particular plugin/configuration pair by name.
5. A way to define logic for "find me the right plugin for X circumstances". (eg, "cache bin for pages". I've been calling this routing.)
6. A way to aggregate plugins together sanely (eg, a View and a Feed are simply collections of defined plugins and configuration. Ideally, text formats would simply become a collection of filter plugins.)
7. All plugins that rely only on context information should, ideally, be auto-cacheable with no further work by the developer. (Blocks will definitely want this, but it also becomes a great built-in cache for Views queries, text filters, etc.)
8. We need a standard way to expose a form to manage configuration, but configuration should be manageable through a script, too, not just a form. (I like the ctools/views plugin model here, except that the configuration object should be a separate object from the plugin itself.)

1 and #2 are such perfect fits for a PHP Interface and a PHP Class that it's not worth discussing other alternatives. The rest are still somewhat open.

To researchers: What we want to know about each system is how it addresses each of the above needs (assuming it does), what's good about it, what's bad about it, and what we could learn from it. For Kohana in particular we should also look into its nested block rendering system, as it seems very close to what we've been discussing.

In practice I suspect we'll still use hooks for much of the definition work. However, one of the high points of the conference was an impromptu meeting with the configuration team, led by David Strauss and Howard Tyson, and Greg Dunlap of deployment fame. One of the things we've been struggling with for Butler is how to store and manage all of this configuration we will need. Most of it requires, given Drupal's current design, either a full database load with lots of queries, lots of really big nested arrays that get cached, or hook invocations that therefore require a full bootstrap. We're trying to avoid all three of those as much as possible.

However, the configuration initiative looks to be moving toward on-disk human-editable JSON files as the primary configuration storage system. That could be loaded trivially simply very very early in the Drupal request lifecycle, and give us read-access to a much richer set of configuration options. If we use that as our primary configuration storage for Context and Plugins, then we can neatly eliminate the most performance-hurting part of the system. It may not help for Drupal 7 backports but it still looks like an extremely promising solution to that problem for Drupal 8.

Some nice supporting links that have come my way in recent weeks for those researching plugin alternatives:

http://components.symfony-project.org/dependency-injection/documentation

http://framework.zend.com/manual/en/zend.controller.actionhelpers.html
http://www.zfsnippets.com/snippets/view/id/48

http://techportal.ibuildings.com/2010/02/22/scaling-web-applications-wit...