Posts tagged CQRS
Just an idea
Feb 3rd
Traditionally web applications are built using some sort of server-side framework like Rails or ASP.NET MVC. Such a framework provide means to implement business logic in a (hopefully) testable manner. Traditionally there are two approaches to shape this logic. One is to leave it close to the UI layer, probably in the Controller part of MVC pattern. The other, advocated by most Domain-Driven Design junkies, is to move it to The Model and leave Controllers as dumb as possible. As you probably already know, I am a proponent of the latter approach. Making Controllers dumb led me to an idea: how about getting rid of the controller and the web framework they are living in and letting client-side JavaScript communicate directly with the domain? Just an idea…
Draft solution
In traditionally approaches where domain entities are moved up and down through all the layers (and almost all tiers) it would be difficult to use them also on client side. That is why we are going to use CQRS as our architectural principle. The first draft solution would look like this:
Client-side JavaScript code is reacting on user input and building commands which are asynchronously sent to the remote facade sitting on the application server. Simple, isn’t it? Is has some drawbacks though.
Problems
First is that client has no means to inspect the outcome of the commands he is sending. He has to duplicate the read model code in order to update the UI widgets. Second, the commands are immediately executed upon reception by the Remote Facade and their outcome is persisted in the Event Store. If a user clicked a wrong button, it would be difficult to undo his action.
Solution, refined
How can we solve this problems? One solution that comes to my mind is to not persist command execution outcome to the event store and read model, but create an in-memory representation of updates to the read model and sent it back to the client. There is no state change on server side. Client can submit commands as user interacts with the UI and receive back the outcome in a denormalized form which can be directly and immediately bound to the UI widgets. All the commands are recorded on client side. The following diagram depicts this process.
When user wants to accept the changes he made, all the recorded commands are send to the server for ‘persistent’ execution.
Challenges
Looks nice, isn’t it? There are some challenges though. There is a lot of JavaScript code to be written to make the client-side plumbing work. There are also some design questions that need to be answered. For example, how do we specify which changes in a read model the client is interested in? Do we send all the changes or somehow ’subscribe’ for particular views in the read model? And how do we create denormalizers so that the same code can be run against both persistent and in-memory read models? We probably need a library for abstracting the read model storage implementation from the denormalizer code. These are hard problems, but I think they are worth solving. The final outcome could be very, very nice.
And it is worth notice that this approach is not limited to the thin client (JavaScript). The same architectural principles should be valid in case of both Silverlight and WPF client.
Compromises
Jan 23rd
Preface
The reality our our profession (mo matter whether you think it’s a craft or a trade or whatever) is that we are not always allowed to make the best and optimal decisions from technical perspective. There are situations when other factors are more important. These factors include things like:
- what skills are there on the team mean to build particular piece of software
- what assumptions and constraints are inherited from previous team who worked on that project
- what client thinks about the solution you propose (yes, unfortunately from time to time clients do influent your technical decisions)
So, lest get to the point. Say you want to pursue the newest and hottest trends in software architecture and want to implement a CQRS/Event Sourcing solution, but one (or more) of things I’ve just listed is preventing you from doing it. You are suck with your favorite ORM for this project (hope it’s not for life). What do you do? Do you give up and build a CRUD solution? I hope you’re not. I’d like to show you one of the alternatives which helped me solve one particular problem.
Problem
Let’s start by asking why do you wanted to use CQRS/Event Sourcing in the first place. Probably because of increased testability, cleaner domain model code and more focused classes. We can achieve pretty good results in this categories by building a CQRS solution with 2 ORM-based models on top of single database. Let me explain the details.
Roles and data flow
The following image depicts the fundamental roles and the data flow in proposed architecture.
User interface is commands to invoke behavior. Commands are processed (since it is a compromise-driven version, command processing is probably synchronous) in the domain model. Because we are using ORM as a persistence engine, I don’t see anything particularly wrong in using Unit of Work pattern to implement collaboration between aggregate roots. After command completes, the whole Unit of Work is persisted by the ORM.
When user interface demands for data, the ORM is used to hydrate (fill with data) a different set of classes, the read model. In pure CQRS scenario, retrieving a data for a view would be a SELECT * operation. Here, since we have only one shared database, we have to create more advanced projections and joins. But that’s the stuff you would have to write anyway if you decided to create a CRUD solution.
Command side
The following diagram shows various roles present on the command side of the equation.
First, there is a command which is sent (as I said, probably synchronously) to the Command Executor object. This guy is responsible for finding a Command Handler for particular command. This is where you could happily use Service Locator and even @kkozmic won’t yell at you:) If he does however, please contact me. But finding a handler is not enough. Command Executor is also responsible for transaction semantics of command execution. It is his job to create a Unit of Work (depending on ORM you’ve chosen, it would be either ISession or some ObjectContext-derived class). Next, he executes the command by passing it to the Command Handler.
First thing that guy has to do is to find domain objects somehow before he’ll be able to trigger their behavior. He does it via Repositories.
Query side
This is way easier. We have some DTOs here that are optimized to transport data to the UI. There DTOs are directly mapped to the database tables. Because there is one shared set of tables for both command and query side, mapping of tables to DTOs is more complicated than in typical CQRS solution. In might involve joins for example.
Finders are for the query side what Repositories are for the command side. Finders, however, doesn’t allow to modify any data.
Quick case study
Here you can download source code of small sample application demonstrating this approach. All you have to do to run it is set up a database and adjust query string in app.config file.
This is a simple banking application. There are two kinds of accounts: internal bank accounts (these does not need to have a balance as they can always be debited or credited) and normal customer accounts (no overdrawn allowed).
The command side uses inheritance to model the nuances in account behavior, while the model on the query side contains one big DTO class for both types of account. The latter decision allows easy binding to the UI controls. It is a quite common pattern to flatten the inheritance structure and represent actual types using enumerated values.
Summary
While I am sure that full-blown CQRS it the approach for solving complex domain problems, it is often very hard to deploy, especially in case of legacy systems. The compromise solution I described may help you in those cases. Once people start seeing real benefits of having some of CQRS principles implemented, it could be easier to convince them to give Event Sourcing a try in the next iteration…
Probably the most powerful CQRS/Event Sourcing platform in .NET
Jan 15th
In the beginning… there were two awesome pieces of code: Ncqrs created by Pieter and Event Store by Jonathan… Now, let me introduce them.
Ncqrs is highly focused on popularizing CQRS and Event Sourcing ideas and, because of that, it is not tied to one particular flavor of CQRS. With Ncqrs you can as well create simple projects backed by RDBMS event store without enforcing aggregate boundaries, as highly scalable reliable solutions without the need of 2PC. The true power of Ncqrs is it’s rich model for representing aggregates. It allows automatic mapping of event handlers using conventions, attributes and via an internal DSL. Ncqrs aggregates can contain entities and nested entities (entities inside entities).
On the other hand, Jonathan’s Event Store is focused on ‘hardcore’ Event Sourcing. The learning curve is high. With Ncqrs you can really write code in the ORMish style. I don’t think you should, but if you can, you have all the facilities, like Unit of Work. You can’t do this with Event Store. This also means that you have smaller chances to create a poorly designed solution. But the feature that makes Event Store a really an awesome tool is support for processing idempotency, integrity and optimistic concurrency based on very loose semantics of most NoSQL databases.
So, why not combine the rich aggregates and asynchronous event processing engine of Ncqrs with storage options (all flavors of SQL, RavenDB, MongoDB), concurrency and idempotency of Event Store? It turned out to be a fairly simple task.
Receiving a command
My first decision was to give up the Ncqrs commanding infrastructure. I wanted to address the scenario where commands are shared resources (e.g. in the enterprise) and they are not allowed to be dependant upon particular framework like Ncqrs (which is an implementation detail of one system). This decision lead me to swap a command service for an interface like this one
public interface IRemoteFacade
{
void Execute(CommandMetadata metadata, Action<AggregateRoot> commandAction);
}
public class CommandMetadata
{
public Guid CommandId { get; set; }
public int? LastKnownRevision { get; set; }
public Guid? TargetId { get; set; }
public Type TargetType { get; set; }
}
Instead of forcing command to implement and interface defined by me (or be decorated by my attributes) so that I can handle them in an automatic way, I acknowledge that commands are autonomous beings and that it is handler’s responsibility to extract necessary metadata from the command object. So, why don’t define ICommandHandler interface like this one
public interface ICommandHandler<TCommand>
{
void Handle(TCommand command);
}
at this level? At first I thought it would be a good idea. Then I realized that this kind of model is already implemented in most transport-level frameworks like NServiceBus. I don’t want to duplicate their functionality. I’d like transport-level handlers (e.g. IHandleMessage<T> in NServiceBus) to call my IRemoteFacade.
Processing the command
After command metadata is extracted and command is handed over to IRemoteFacade, the real processing begins. First, a new instance of aggregate of specified class is created. Then, if the command is targeted against an existing aggregate, an event stream consisting of all events from the latest snapshot up to the last known revision is retrieved. If no revision is provided in the command (which means that optimistic concurrency is disabled for particular command), event are retrieved until most recent commit.
The state of the aggregate is reconstructed using retrieved events. Then, the action provided in Execute call is invoke against the aggregate object.
The last (but certainly not least) activity is saving the state of the aggregate. Events resulting from processing the command are retrieved from the aggregate object, transformed and pushed into the Event Store. If there were optimistic concurrency violations, exception will be thrown. Also, if Event Store detected duplicate command it will signal this by throwing an exception. It is up to IRemoteFacade caller to catch this exception and (most probably) swallow it.
Integration with NServiceBus
All this stuff would be useless if commands cannot be send to the IRemoteFacade. There has to be some transport mechanism. While I don’t want to tie to a single transport, I want to support some common (in my opinion) scenarios. I decided to add NServiceBus support first.
NServiceBus has a notion of message handlers defined as IHandleMessages<T> interface. OOTB you would have to code the facade calling code by hand. To avoid duplication I prepared a special base class:
public abstract class CommandHandlerBase<TMessage, TAggregateRoot> : IHandleMessages<TMessage>
where TMessage : IMessage
where TAggregateRoot : AggregateRoot
{
public IRemoteFacade RemoteFacade { get; set; }
public IBus Bus { get; set; }
public virtual void Handle(TMessage message)
{
var metadata = ExtractMetadata(message);
if (metadata.TargetType == null)
{
metadata.TargetType = typeof (TAggregateRoot);
}
try
{
RemoteFacade.Execute(metadata, x => Handle(message, (TAggregateRoot)x));
}
catch (DuplicateCommitException ex)
{
}
}
protected abstract CommandMetadata ExtractMetadata(TMessage message);
protected abstract void Handle(TMessage message, TAggregateRoot aggregateRoot);
}
The code is straightforward. All you have to do is create your own code for extracting command metadata (or its envelope) and for invoking particular action on the aggregate root. The Handle method is also a good place to put command validation logic. Now, how can I tell NServiceBus to use all this infrastructure? It is as simple as calling one method:
_bus = Configure.With()
.Log4Net()
.DefaultBuilder()
.XmlSerializer()
.MsmqTransport()
.IsTransactional(true)
.PurgeOnStartup(true)
.UnicastBus()
.LoadMessageHandlers()
.InstallNcqrs()
.UseInMemoryPersistenceEngine()
.CreateBus()
.Start();
The additional UseInMemoryPersistenceEngine method instructs Event Store which storage engine to use.
Summary and what’s next
By combining the strongest points of Ncqrs and Event Store I believe I created the most powerful and universal tool for implementing CQRS in .NET platform. Add NServiceBus for command transport and you have a complete end-to-end solution.
The source code can be downloaded from my fork of Ncqrs on github. It is still a prototype but you can see the power it brings.
The next goal is to integrate it smoothly with asynchronous event processing engine of Ncqrs.
The theory of Event Sourcing and the theory of EDA
Sep 29th
Events are key concept in two recently very rapidly evolving areas of software development, namely Event Sourcing and Event Driven Architectures (EDA). I bet most of us have a gut feeling that both of these event-centered ideas have something in common, but there is little quality information on that subject. I would like to publish my opinions on that matter in this post. Other very interesting opinions can be found here and here.
There were some attempts to unify the ‘theory of events’ and they resulted in discussions that, in my opinion, lead to no conclusion. This is because we all come from different backgrounds and little differences in understanding some of the concepts add one to another and, in the effect, cause big confusion. Here is what I propose: let’s start with dividing the subject area into three distinct contexts: web-scale applications, business systems and whole enterprises.
Web-scale applications
The primary reason for using Event Sourcing in context of web-scale applications is, well, scale. Event Sourcing scales very, very well. In such systems we can have 10s, 100s or even 1000s of partions, each one having all the tiers and only a part of the data. The logic of application is, relatively, simple. And there is of course only one application which, of course, can interact with other systems, but this is a marginal problem here.
I believe Rinat Abdullin was writing about such a system. His great and very detailed article can be (at least I believe it can) summarized be following interaction patterns:
- Message (command-style) -> AR -> Event
- Message (event-styte) -> Saga -> Event
- Event -> Event Subscription -> Message
I’ve just introduced a concept of Saga here. Let’s define it in the context of stand-alone web-scale system. Saga is a coordinator for actions involving multiple Aggregate Roots. Saga’s responsibilities are, among others, resolving problems with lost or out-of-order messages. Given that most of web-scale systems contain fairly simple business logic, I agree with Rinat that the complexity of such a system is localized mainly in sagas, not in the ARs (which, as he said, mostly play houskeeping role).
Starting with a UI-issued command, the command message is passed (via a router) to the proper partition where it is passed to a concrete Aggregate Root. In the effect the aggregate root publishes an event which is stored locally and is (also locally) checked against all defined Event Subscriptions (which are fairly static and can be easily shared among all the partitions). If an event subscription applies, a resulting message (which have a concrete destination being either AR or a Saga) is send to the router.
The same rules apply if a message is destined to a Saga. Instances of both Sagas and ARs live in only one partition. Transformation of events to messages done by Event Subscription is essential to avoid the necessity to publish all events to all partitions (which is impossible given such a large potential number of partitions).
The new and shiny concept of CEP (Complex Event Processing) can also be introduced to such a system. CEP can be used for implementing ’smart’ Event Subscriptions.
Business aplications
There applications are workhorses of each IT-enabled company. This group consists of all kinds of business systems such as CRMs, ERPs, order processing apps and so on and so forth.
These systems usually doesn’t have very high performance requirements. Most of them doesn’t need data partitioning at all and even if so, the number of partitions is less than 10. The reason for using Event Sourcing in such systems is obviously not the scaling capability. The reasons can be various, but usually are one or more of the following:
- proven audit trial (regulatory requirements)
- ability to inspect object state as it was in any moment in time
- improved testability or maintainability
Interaction paths in such a system are, not surprisingly, quite different from a web-scale app:
- Command -> AR -> Event
- Event -> Saga -> Command
- Event -> Event Subscription -> Enterprise-Level Event
- Event -> CEP (Enterprise-Level)
As previously, starting with UI interaction, a command is generated and passed to the system where it is routed to the Aggregate Root. As the result of processing the command, one or more events are generated by the AR and passed to the event storage. In a business application, while there can be more than one copy of the business tier (processing logic), there is usually only one instance of event storage for a given Aggregate Root. Because of that, Sagas can be directly fed with events (without the need of explicit Event Subscriptions).
In a single business application the main responsibility of Sagas is coordination between ARs. Given that there is no partitioning, there can also be no out-of-order or lost messages. The result is small and focused sagas containing almost no logic at all. All the complex logic (such as computations) is placed in Aggregates.
Events generated by single ARs in a single business application can affect the whole enterprise. Depending on actual needs and maturity of EDA architecture in the organization, AR-issued events can be passed to the enterprise level in two ways.
- explicitly, by using a concept of Event Subscription which translates one or more low-level AR events to one enterprise-level event. There high-level events are shared by all systems of the organization and are preferable governed by some central authority.
- implicitly, by forwarding them to the enterprise-level CEP infrastructure. Advanced CEP capabilities are then used to detect various patterns in the event stream.
Enterprise-level architecture
Things look a little bit different at the enterprise level. The main difference is we don’t have Aggregate Roots here. We have whole systems instead. Because of that, we can’t say we use Event Sourcing — we apply Event Driven Architecture (EDA). To avoid confusion of high level events, which are used here, with low level ones, I will refer to the former as Enterprise-Level Events (EL Events). The interaction paths can be summarized as:
- Command -> System -> EL Event
- EL Event -> Saga -> EL Event
- Event -> CEP -> EL Event
- EL Event -> CEP -> EL Event
- EL Event -> Event Subscription -> Command
The first notable similarity between enterprise and previous two contexts is the recurring concept of Saga. It looks like this term is highly overloaded these days. In the context of an enterprise, a Saga reacts on EL Events by producing other EL Events. It usually don’t have to deal with lost messages, as this issue is resolved by the infrastructure. It must, however, deal with out-of-order messages.
A system reacts on a command passed to it. In the result, a system can (but doesn’t have to) generate an EL Event (via system-level event subscription). System-level events can also (which was stated before) routed as-is to the enterprise level where they are fed to the CEP infrastructure. As a result, a EL Event can be generated.
EL Events themselves can also be fed to CEP resulting in another EL Events coming out of CEP.
Finally, EL Events can be, be means of Event Subscriptions, transformed to Commands destined to concrete system and concrete Aggregate Root inside that system.
Summary
While all three contexts emphasize the usage of events and make use of sagas, these terms have different meaning and purpose in each one. Therefore, there can’t ever be a single model that is suitable to all of them. The moral is easy to predict: context is king.
Ncqrs
May 30th
W ciągu ostatnich dwóch tygodni moją uwagę przykuł na dobre nowy framework open source — Ncqrs (witryna CodePlex Ncqrs znajduje się tutaj). Jak sugeruje nazwa, Ncqrs służy do budowy systemów w oparciu o wzorzec architektoniczny Command-Query Responsibility Separation (CQRS). To, czego nazwa nie mówi, to fakt, że Ncqrs narzuca pewną specyficzną implementację wspomnianego wzorca, a mianowicie tę opartą o technikę Event Sourcing. Na podstawie posta Grega Younga można by się czepiać, że nazwa Ncqrs nie jest zbyt trafna, ale odłóżmy kwestie nomenklatury na bok. Czym jest Ncqrs i jak to się stało, że mnie tak zafascynował?
Zasada działania
Ncqrs jest całościowym rozwiązaniem służącym do budowy systemów opartych o silny model domeny, którego stan jest przechowywany za pomocą strumienia zdarzeń. Poniższy diagram prezentuje workflow dla pojedynczego przypadku użycia w Ncqrs.
Komendy
Punktem wejścia do Ncqrs są komendy. Są to obiekty, który reprezentują żądania wykonania pewnej operacji na modelu domeny. Komendy są mapowane na operacje za pomocą rozbudowanego rozszerzalnego mechanizmu. Najprostsza implementacja mappera opiera się na dwóch atrybutach, które określają, czy dana komenda ma wykonywać metodę istniejącego obiektu (w nomenklaturze Ncqrs — korzenia agregatu), czy też tworzyć nowy obiekt. Properties komendy są mapowane (na podstawie nazwy) do parametrów wybranej metody lub konstruktora. Ostatecznie, (jeśli to konieczne) z magazynu danych podnoszony jest odpowiedni obiekt i wykonywana jest odpowiednia metoda (lub konstruktor).
Operacje biznesowe
Operacje i konstruktory obiektów są w Ncqrs wywoływane tylko za pośrednictwem komend. Ich jedynym zadaniem jest wykonanie logiki biznesowej. Nie mogą one bezpośrednio modyfikować stanu obiektu. Zamiast tego, dozwolonym mechanizm modyfikacji stanu jest zgłaszanie zdarzeń. Operacje biznesowe mogą także komunikować się ze światem zewnętrznym.
Stosowanie zdarzeń
Jak już wspomniałem na wstępie, zdarzenia są sposobem przechowywania stanu obiektów w Ncqrs. Operacja biznesowa może zgłosić jedno lub więcej zdarzeń. Dla każdego z nich framework wyszukuje odpowiedniej metody je przetwarzającej. Mechanizm ten jest oczywiście rozszerzalny, a out-of-the-box Ncqrs zapewnia dwa sposoby wiązania zdarzenia z metodą obiektu biznesowego przeznaczoną do jego obsługi: za pomocą konwencji oraz za pomocą atrybutów. Ncqrs wywołuje znalezioną metodę przekazując jej zgłoszone zdarzenie.
Przetwarzanie zdarzeń
Metoda przetwarzająca zdarzenie modyfikuje stan obiektu na podstawie danych przekazanych zdarzeniu. Tylko tyle i aż tyle. Metoda ta nie powinna zawierać żadnej logiki biznesowej (warunkowej), ani mieć jakichkolwiek skutków ubocznych (komunikacja z innymi systemami itp.).
Zapewne chcielibyście zapytać po co tyle komplikacji? Dlaczego operacja biznesowa nie może zmodyfikować stanu? Odpowiedź jest prosta. Ponieważ stan obiektów jest reprezentowany jako strumień zdarzeń przez nie wygenerowany, aby odtworzyć obiekt niezbędne jest stworzenie jego pustej instancji, a następnie przetworzenie (w kolejności!) wszystkich zapisanych zdarzeń — oczywiście za pomocą odpowiednich metod przetwarzających. Metody te są więc nie tylko stosowane do modyfikacji stanu podczas przetwarzania, ale także do odtwarzania tego stanu podczas podnoszenia obiektu z trwałego magazynu.
Dzięki takiemu podejściu systemy Event Sourcing (jak Ncqrs) zapewniają, za darmo, ślad audytowy, który ma gwarancję poprawności, ponieważ jest on wykorzystywany do budowy obiektów podczas normalnego działania systemu.
Publikowanie i denormalizacja zdarzeń
Jeśli wszystko do tej pory przebiegło prawidłowo, wszystkie zgłoszone zdarzenia są publikowane. Oczywiście, także w tym wypadku Ncqrs pozwala wymienić mechanizm publikacji zdarzeń. Domyślny wykorzystuje komunikację wewnątrz procesu, ale dostępny jest także taki, który wykorzystuje NServiceBus.
Publikowanie zdarzeń ma dwa cele. Po pierwsze, pozwala powiadomić zainteresowane systemy zewnętrzne o zmianach stanu naszego systemu. Polega to na eskalowaniu “lokalnych” (dotyczących naszego systemu) zdarzeń do statusu zdarzeń “globalnych” (mających znaczenia dla całego środowiska systemów). Stąd już tylko jeden krok do pełnej Event Driven Architecture (EDA).
Drugim celem publikowania zdarzeń jest tzw. denormalizacja, czyli aktualizacja podsystemu obsługi zapytań. Jaki podsystem? O co chodzi? Dokładny opis zagadnienia CQRS znajduej się tutaj. W tym miejscu wspomnę tylko, że systemy CQRS wykorzystują zwykle dwa osobne magazyny danych dla przetwarzania komend oraz do realizacji zapytań. Do synchronizacji tego drugiego magazynu danych wykorzystywane są właśnie denormalizatory zdarzeń. Proces denormalizacji polega na wykonaniu w bazie danych dla zapytań modyfikacji, które wynikają z opublikowanego zdarzenia. Skąd taka nazwa? Otóż zdarzenia stanowią znormalizowaną (pozbawioną redundancji) postać danych. W magazynie dla zapytań zaś, te same dane mogą mieć wiele reprezentacji, ponieważ nadrzędnym celem jest optymalizacja czasu realizacji zapytań.
Dlaczego to może działać?
Jest kilka powodów, które sprawiają, że (pozornie) szalona idea reprezentacji stanu obiektów jako ciągu zdarzeń może działać w praktyce. Oto kilka z nich:
- zapewnia darmowy, gwarantowany, ślad audytowy
- możliwość wykonywania tzw. snapshot’ów (czyli pełnych zrzutów zserializowanego obiektu biznesowego) co N zdarzeń. Dzięki temu odtwarzanie obiektu wymaga jedynie przetworzenia zdarzeń, które nastąpiły po ostatnim snapthot’cie
- w klasycznym DDD (z użyciem O/RM) podczas podnoszenia obiektu z bazy danych tak naprawdę pobierane jest wiele wierszy danych (np. za pomocą podzapytań i złączeń). W przypadku Event Sourcingu pobieranych jest kilka wierszy przechowujących zdarzenia. Oba podejścia mają więc podobną złożoność na poziomie bazy danych.
Zapraszam Was do zabawy z Ncqrs. Postaram się odpowiedzieć na wszystkie Wasze pytania.
