The Core Initiative Trinity

The twin deep-engineering Core Initiatives of Configuration and Web Services have been taking a long hard look at Drupal's basic, underlying architecture of late. We're swirling around three fundamental pieces that should, we believe, revolutionize Drupal core and Drupal development, allowing for a quantum leap in Drupal's capabilities. (Excited yet?) The catch is, we have to get these right. More than that, we need to nail these. For that reason, we're trying to solicit wider feedback on a few key architectural questions. I have tried to summarize the situation and the challenges below, and invite informed, insightful feedback from the community.

The three cornerstones of this new approach are Configuration, Context, and Plugins. All three are things that Drupal core does not, in fact, provide in anything resembling a useful way. We need to get all three of them into core, however, and they need to work together well. The problem is that there are very good reasons for all of them to require one of the others first, which creates a circular dependency. We need to sort out that problem before we can get building.

Configuration

At present, Drupal core offers only one configuration system: variable_get(). That is, the only configuration system available in core is a single, inefficient, unsearchable key/value store. All other forms of configuration in Drupal, even in core itself, are one-off implementations, most of which use the database directly because, well, it's there. That has led to the current situation where managing configuration in anything resembling a sane fashion is ugly, a hack, or both. "Exportables" are currently a widely popular solution, but are still backward because it puts the database as the primary source of information and we still have no standardization whatsoever for those systems.

The Configuration and Deployment initiative, headed by Greg Dunlap, is looking to solve this issue by replacing variable_get() with a more robust and extensible in-core configuration system that stores data primarily on disk, which makes it a hundred times easier to version control, deploy, and manage. And by offering more features than variable_get(), we can get rid of most or all of our one-off configuration systems and fold them down into a single, powerful, deployable system. By basing it on the file system, we can make it available at an extremely low level very early in the request cycle.

For more information, see Greg's battle plan writeup.

Context

Drupal core currently offers almost no mechanism for tracking or handling context. By context, we mean useful information that makes one request different from another; the requested URL, the objects specified by that URL, the current user, the current language, the type of data the user agent (usually the web browser) wants back, what section of the site we're in, etc. Every single one of those pieces of information may be extremely important, but every single one of them is currently handled in a completely different way. The closest we come to handling context is the menu load arguments passed to menu callbacks, which derive from the URL. For pretty much everything else you can think of, you're totally on your own. And don't even think about trying to properly unit test that.

The Web Services and Context initiative, headed by Larry Garfield, aims to introduce a unified context broker that can be used system-wide. By replacing a hundred one-off utility functions that don't talk to each other and are all simply a wrapper around global state with a clean, unified system for taking requests for contextual information and mapping those to the proper code to determine that information, we can streamline the way Drupal responds to incoming data. It also will allow us to more cleanly segregate parts of the system while still allowing them to share important information, which simultaneously gets us better unit testing and makes it easier for different systems to integrate with each other.

For more information, see the WSCCI definitions page.

Plugins

Drupal core currently has no system for plugins at all. Plugins are not hooks; hooks are event handlers, in essence. Plugins are configurable, swappable implementations of some task. That could be alternate cache backends (currently classes with a common interface and hard-coded $conf settings); it could be alternate session storage (currently swappable include files and hard-coded $conf settings); it could be blocks (currently a one-off SQL table that no one really likes); it could be Views (currently a named, nested set of custom objects and arrays with a custom serializer); it could be text formats (currently an info hook that then gets stored in a totally custom way); it could be Field widgets (currently pseudo-hook callbacks that are architecturally backward); it could be the type of HTTP response to send (currently not pluggable at all, which is a problem). Every single one of those is its own one-off implementation. And most of them suck.

The Web Services and Context initiative also aims to introduce a single, in-core, robust plugin system to unify all of those systems and more. By having a single, robust system we can vastly reduce the amount of code in core, we can vastly reduce the complexity of core making it harder to introduce bugs, we can vastly simplify core's API so that there is only a single system for new developers to learn, and we can make pluggable things that currently are not. That opens the doors for fascinating concepts like swapping out implementations of core logic, moving Drupal to an entirely non-SQL environment, substantially more robust handling of web services and REST requests, and other fun things.

Note: Plugins are not intended to replace hooks; they complement them by solving a problem that core doesn't currently solve and for which hooks are an extremely poor fit.

For more information, see the WSCCI definitions page.

The problem

Those three key changes are fundamental to Drupal, but are critically important. So what's the problem?

• The problem is how those three pillars relate.
• We want the configuration system to be flexible, and allow for more than just one hard-coded implementation. That means it needs plugins.
• We need the plugin system to allow plugins to be configurable. That means it needs configuration.
• We need the plugin system to be able to respond based on more robust request information. That means it needs context.
• We need the context system to support swapping out what code is responsible for handling what piece of contextual information. That means it needs those handlers to be pluggable.
• We need to be able to configure what code is responsible for handling what piece of contextual information. That is, it needs configuration.

That creates a horrible mess of circular dependencies, and we're dead before we even start. We have to resolve the relationship between these three systems before we can put pen to paper (or finger to keyboard, as it were).

Possible solutions

A number of possible ways around this problem have been bandied about, but not yet settled on. I have attempted to document the main ones below. (Note that these are not mutually exclusive.)

1. Don't make configuration storage pluggable. Just accept that there will be one (1) way that configuration gets stored and be done with it. Possibly we could allow only certain parts of configuration to be pluggable, those that are not needed early on.

   • Pros: Eliminates the dependency on plugins, simplifies the implementation, and ensures that configuration is the baseline system upon which everything can be built.
   • Cons: Little to no flexibility. If you have multiple web heads then making changes via files on disk are actually harder to propagate than database changes.

2. Implement a simpler, more streamlined pluggable mechanism for context handlers other than plugins.

   • Pros: Eliminates the dependency on plugins. Allows for a finely-tailored light-weight system in a very performance-sensitive area.
   • Cons: We're back to having multiple pluggable mechanisms, even if we only plan on the one one-off.

3. Implement a simpler, more streamlined pluggable mechanism for configuration other than plugins.

   • Pros: Eliminates the dependency on plugins. Allows for a finely-tailored light-weight system in a very performance-sensitive area.
   • Cons: We're back to having multiple pluggable mechanisms, even if we only plan on the one one-off.

4. Allow context to be optional for plugins, based on their type.

   • Pros: Plugin types that don't need context don't have to deal with them. We can use plugins for context handlers because we then don't have a circular dependency, so there's more consistency.
   • Cons: Adds complexity to the plugins system, since there are multiple code paths. You may define a plugin type to not need context, but then oops, one particular implementation may need it and then you need to try and hack your way out of that hole.

5. Allow configuration to be optional for plugins, based on their type.

   • Pros: Plugin types that don't need configuration don't have to deal with them. We can use plugins for configuration, because then we don't have a circular dependency, which allows configuration itself to be swappable.
   • Cons: Adds complexity to the plugins system, since there are multiple code paths. You may define a plugin type to not need configuration but then oops, one particular implementation may need it. (We saw this with field formatters in Drupal 6.) Still doesn't solve the problem of how you configure which plugin to use for configuration.

And probably others. We need to settle on some combination of these trade-offs in order to properly implement these three key systems.

Proposals

There are two main proposed approaches so far, which I will try to adequately summarize.

In proposal A, we basically do #1 and #2 above (or possibly #2 and #3). That is, configuration is the bottom of the stack. It depends on nothing except having a filesystem that it can access (either via fopen() or SplFile) and the PHP JSON libraries (which have been standard on PHP installs for years now). We then use configuration and a custom one-off system to build our context handling system. We then use configuration plus the context system to build a plugin system. Plugins then always take context and configuration, consistently. If a given plugin implementation has no need for one or the other it simply ignores it, but it's there just in case. If we decide that configuration handling needs to be pluggable, we do that with a custom, low-level approach as well.

In proposal B, we do #4 and #5. Plugins, then, become the fundamental underlying layer. We eat the extra complexity of a single plugin system with multiple usage modes, and then use plugins in "reduced mode" to power both swappable configuration storage systems and the mapping for context handlers. Configuring those early plugin types is done via custom PHP code.

And there may be other combinations we're not fully exploring yet, but need to.

Note that all of this work is happening very early in the bootstrap process, so it must all be fast and Drupal's usual hammer, "when in doubt add a hook", is not available to us at all.

RFC

At this point, we'd like the thoughts of the rest of the community on how to move forward. Every option has significant trade-offs, and we need to decide which trade-offs we want. This may be one of the most important decisions we collectively made for Drupal 8.

Please keep comments on topic and constructive. :-) Let the bikeshedding commence.