Posts tagged event sourcing
Events are not an implementation detail
Nov 29th
When listening to many discussion about Domain-Driven Design and Event Sourcing, subjects which I am recently very into, sometimes I have a feeling that events are considered merely a way of persisting aggregates. People tend to discuss benefits events have with regards to storing historical data, having possibility to generate any read model out of events and so on. Reducing event sourcing to such ‘housekeeping’ duties is castrating your architecture.
This point of view is expressed in a way people model their event-based solutions: thinking first about the aggregates and then, within each aggregate, about events they generate. Sometimes the process is even more event-UNcentric when people first implement a command and then start thinking about what event types should be created.
Given the fact that most Event Sourcing practitioners have a background in DDD, we tend to bring our experience from working in data-centric environment to the event-centric world. There is, however, a fundamental difference between these two.
When building a behaviour-oriented DDD solution on top of a data-centric infrastructure (a database) you do your best to hide the database and its schema from the business layer. Over the years we learnt that it is better treated as implementation detail because it does not provide concepts useful to model behaviour. There concepts must be built on top of it. The underlying data layer must not be visible to the outside world because of it’s volatility — it has to change whenever the behaviour it supports needs to change.
On the other hand, when building a similar solution on top of event-centric infrastructure you can (and should) leverage events to model the behaviour. They (not aggregates or anything built on top of them) are the central concept here. Whenever business changes, new event types appear and some old are no longer generated but no event is ever lost. Whatever happened, happened.
Returning to my original thought, in an event-centric solution aggregates should not be treated as first-class citizens. Don’t get me wrong, they are very useful. In a healthy design probably a vast majority of events would be generated via aggregates, but some would not. Aggregates are priceless when it comes to modelling concurrency and to encapsulating business logic.
You may now want to ask a question when should you use aggregates and when not? The answer seems to be simple. Whenever your system needs to respond to some external activity, you just directly inject an event representing what have happened into your system. The idea is, again, whatever happened, happened. You can’t throw concurrency exception at The World saying it is wrong to raise this event right now. You have to humbly acknowledge the fact and record the information. On the other hand, when it is your system who conducted the activity to be communicated by an event, you should use an aggregate to have full control over how and when the event is published.
Have fun building event-centric solutions!
Testing strategies for business logic
Sep 1st
Let’s talk about testing. How do you test your business logic? I spent some time thinking about it and came up with three different strategies for three different ways of implementing business logic.
CRUD
As you can read here, I think CRUD is OK for small and simple applications. You have to, however, remember what is not supported in this architecture – collections of nested objects.In the previous post I’ve written about few reason why is that. Today I will show you another one, this time related to testing. You test a CRUD application by performing a business action given some fake repository object(s) and verifying (on this fake repository) if proper objects have been created, updated and deleted.
//Arrange var orderRepo = new FakeOrderRepository(); var orderService = new OrderService(orderRepo); //Act orderService.CreateNewOrder(orderData); //Assert var order = orderRepo.Get(id); Assert.IsNotNull(order); Assert.AreEqual(10, order.Value);
That’s it, no rocket science. It would be much more difficult, however, if you allowed relations between objects. In such case you would have to verify also these relations. Moreover, mocking a repository if you support relations would be much harder. You would basically had to provide an in-memory database. If you stick to simple structures, you mock a repository using a single collection and your’re done.
The classic approach to DDD
This approach is basically a compilation of what you can find in the Blue Book and in Jimmy Nilsson’s book. It is Domain-Driven Design with objects mapped to database using an Object-Relational Mapper (remember — NHibernate). These objects have properties for getting their attributes and showing them on the UI. Of course there is no dumb setters. Last but not least, there are also value objects that are used to represent attributes with complex values.
So how do you test such architecture? You take advantage of persistence ignorance of good domain model. Prepare a domain object in state you want to test by invoking normal business methods. Once it is ready, call the method you want to test. Then verify the results by using property getters.
//Arrange
var customer = new Customer("Szymon", "Pobiega");
var product = new Product("Widgets", new Money(10, new Currency("USD")));
var order = new Order(orderId, customer);
order.AddLineItem(3, product);
//Act
order.AddLineItem(1, product);
//Assert
Assert.AreEqual(2, order.LineItems.Count())
Testing value objects is even simpler because of their immutability. All you need is to call a method on it (returning a new instance of a value object) and compare it with expected value.
//Arrange
var usd = new Currency("USD");
var total = new Money(10, usd);
var toSubtract = new Money(2, usd);
//Act
var result = total.Minus(toSubtract);
//Assert
var expected = new Money(8, usd);
Assert.AreEqual(expected, result);
The trick in testing classic DDD is to not depend on your ORM to do anything for you behind the scenes. The good example is many-to-many relation. In NHibernate, if you set value only on one end and persist the entities, both ends will be set magically when you get the object from database next time. But before that, for some time the object model will be inconsistent. Good practice for such relations is have code in domain model that explicitly sets both ends of particular relation.
Another gotcha is preparing the object state. Like I said, you do it by calling normal methods. The downside of this approach is something may go wrong in this phase of test and the tested method will be given different state than expected. My personal pragmatic approach is to do nothing until it hurts. If it starts to cause major problems you can think about adding Assert calls after the object has been prepared to verify if it really is in the expected state.
Event Sourced DDD
One of the reasons I really like Event Sourcing is it doesn’t cause any (known to me) problems when testing. If I was to name the ideal types of code for testing, the first one on the list would be functional code and event sourced entities would be right behind it.
The beauty of testing event sourced objects comes from the fact that state mutations and business logic are separated. You can mutate the state (by applying an event) without any business action. That’s how entities are hydrated from the event stream in the first place. There is also an inherent beauty in definition of such tests
//Arrange var order = new Order(); order.Apply(new ItemAddedEvent(10, productId)); order.Apply(new ItemAddedEvent(5, productId)); order.Apply(new ItemRemovedEvent(2, productId)); //Act order.AddItem(3, productId); //Assert order.UncommittedEvents.Contains(new ItemAddedEvent(3, productId));
Which translates to
Given these events had happened before, when this action is called, I expect these events to be published
Nice, isn’t it?
Summary
Of course you can’t always afford to create a fully-featured event sourced solution. For simple scenarios it’s an overkill. But that’s not an excuse for not having good tests. Every approach to structuring your business logic is testable, you only have to know what constraints you have to place on the code so that testing it is a pleasure, not a pain in the back.
Probably the most powerful CQRS/Event Sourcing platform in .NET
Jan 15th
In the beginning… there were two awesome pieces of code: Ncqrs created by Pieter and Event Store by Jonathan… Now, let me introduce them.
Ncqrs is highly focused on popularizing CQRS and Event Sourcing ideas and, because of that, it is not tied to one particular flavor of CQRS. With Ncqrs you can as well create simple projects backed by RDBMS event store without enforcing aggregate boundaries, as highly scalable reliable solutions without the need of 2PC. The true power of Ncqrs is it’s rich model for representing aggregates. It allows automatic mapping of event handlers using conventions, attributes and via an internal DSL. Ncqrs aggregates can contain entities and nested entities (entities inside entities).
On the other hand, Jonathan’s Event Store is focused on ‘hardcore’ Event Sourcing. The learning curve is high. With Ncqrs you can really write code in the ORMish style. I don’t think you should, but if you can, you have all the facilities, like Unit of Work. You can’t do this with Event Store. This also means that you have smaller chances to create a poorly designed solution. But the feature that makes Event Store a really an awesome tool is support for processing idempotency, integrity and optimistic concurrency based on very loose semantics of most NoSQL databases.
So, why not combine the rich aggregates and asynchronous event processing engine of Ncqrs with storage options (all flavors of SQL, RavenDB, MongoDB), concurrency and idempotency of Event Store? It turned out to be a fairly simple task.
Receiving a command
My first decision was to give up the Ncqrs commanding infrastructure. I wanted to address the scenario where commands are shared resources (e.g. in the enterprise) and they are not allowed to be dependant upon particular framework like Ncqrs (which is an implementation detail of one system). This decision lead me to swap a command service for an interface like this one
public interface IRemoteFacade
{
void Execute(CommandMetadata metadata, Action<AggregateRoot> commandAction);
}
public class CommandMetadata
{
public Guid CommandId { get; set; }
public int? LastKnownRevision { get; set; }
public Guid? TargetId { get; set; }
public Type TargetType { get; set; }
}
Instead of forcing command to implement and interface defined by me (or be decorated by my attributes) so that I can handle them in an automatic way, I acknowledge that commands are autonomous beings and that it is handler’s responsibility to extract necessary metadata from the command object. So, why don’t define ICommandHandler interface like this one
public interface ICommandHandler<TCommand>
{
void Handle(TCommand command);
}
at this level? At first I thought it would be a good idea. Then I realized that this kind of model is already implemented in most transport-level frameworks like NServiceBus. I don’t want to duplicate their functionality. I’d like transport-level handlers (e.g. IHandleMessage<T> in NServiceBus) to call my IRemoteFacade.
Processing the command
After command metadata is extracted and command is handed over to IRemoteFacade, the real processing begins. First, a new instance of aggregate of specified class is created. Then, if the command is targeted against an existing aggregate, an event stream consisting of all events from the latest snapshot up to the last known revision is retrieved. If no revision is provided in the command (which means that optimistic concurrency is disabled for particular command), event are retrieved until most recent commit.
The state of the aggregate is reconstructed using retrieved events. Then, the action provided in Execute call is invoke against the aggregate object.
The last (but certainly not least) activity is saving the state of the aggregate. Events resulting from processing the command are retrieved from the aggregate object, transformed and pushed into the Event Store. If there were optimistic concurrency violations, exception will be thrown. Also, if Event Store detected duplicate command it will signal this by throwing an exception. It is up to IRemoteFacade caller to catch this exception and (most probably) swallow it.
Integration with NServiceBus
All this stuff would be useless if commands cannot be send to the IRemoteFacade. There has to be some transport mechanism. While I don’t want to tie to a single transport, I want to support some common (in my opinion) scenarios. I decided to add NServiceBus support first.
NServiceBus has a notion of message handlers defined as IHandleMessages<T> interface. OOTB you would have to code the facade calling code by hand. To avoid duplication I prepared a special base class:
public abstract class CommandHandlerBase<TMessage, TAggregateRoot> : IHandleMessages<TMessage>
where TMessage : IMessage
where TAggregateRoot : AggregateRoot
{
public IRemoteFacade RemoteFacade { get; set; }
public IBus Bus { get; set; }
public virtual void Handle(TMessage message)
{
var metadata = ExtractMetadata(message);
if (metadata.TargetType == null)
{
metadata.TargetType = typeof (TAggregateRoot);
}
try
{
RemoteFacade.Execute(metadata, x => Handle(message, (TAggregateRoot)x));
}
catch (DuplicateCommitException ex)
{
}
}
protected abstract CommandMetadata ExtractMetadata(TMessage message);
protected abstract void Handle(TMessage message, TAggregateRoot aggregateRoot);
}
The code is straightforward. All you have to do is create your own code for extracting command metadata (or its envelope) and for invoking particular action on the aggregate root. The Handle method is also a good place to put command validation logic. Now, how can I tell NServiceBus to use all this infrastructure? It is as simple as calling one method:
_bus = Configure.With()
.Log4Net()
.DefaultBuilder()
.XmlSerializer()
.MsmqTransport()
.IsTransactional(true)
.PurgeOnStartup(true)
.UnicastBus()
.LoadMessageHandlers()
.InstallNcqrs()
.UseInMemoryPersistenceEngine()
.CreateBus()
.Start();
The additional UseInMemoryPersistenceEngine method instructs Event Store which storage engine to use.
Summary and what’s next
By combining the strongest points of Ncqrs and Event Store I believe I created the most powerful and universal tool for implementing CQRS in .NET platform. Add NServiceBus for command transport and you have a complete end-to-end solution.
The source code can be downloaded from my fork of Ncqrs on github. It is still a prototype but you can see the power it brings.
The next goal is to integrate it smoothly with asynchronous event processing engine of Ncqrs.
Why do I find Event Sourcing interesting?
Jun 7th
What makes Event Sourcing so interesting? I it is not the free, proven, audit trial, nor the possibility of reincarnating object in any particular state it had in the history. It is also not the great performance of add-only event store. It is testability.
Why Event Sourcing makes things testable? By splitting up logic responsible for two distinct things: calculating new state of the object and applying this state change. Normally, these two operations are mixed in one business logic method like this:
public class Customer
{
public void PlaceOrder(Product product, int quantity)
{
CheckProductAvailability();
CheckCustomerCreditLimit();
CheckSomethingElse();
decimal price = CalculateTotalPrice(product, quantity);
Order o = new Order(product, quantity, price);
_orders.Add(o);
}
//...
}
If you read this carefully, you will see that first 4 lines contain the business logic of state change calculation. The last 2 lines, on the other hand, contain no logic at all. Now, what’s the problem with this code’s testability?
The problem becomes visible, when I want to test how PlaceOrder behaves when adding fifth order. I have to call PlaceOrder four times to prepare state of the object. That is wrong, because my test code invokes a lot of logic during preparation phase. This means that a lot of things can go wrong and nobody likes tests which fail before the actual test code gets executed.
Let’s see if PlaceOrder can be corrected using SOLID principles. Does it have exactly one reason to change? It seems like it does. When order placement logic changes, the method changes, right? Let’s dig deeper. What about the last two lines? Do they change when the rest of method changes? Well, I don’t think so. Looks like they indeed do represent a different concern. So let’s refactor them to a separate method.
public class Customer
{
public void PlaceOrder(Product product, int quantity)
{
CheckProductAvailability();
CheckCustomerCreditLimit();
CheckSomethingElse();
decimal price = CalculateTotalPrice(product, quantity);
ApplyPlaceOrder(product, quantity, price);
}
private void ApplyPlaceOrder(Product product, int quantity, decimal price)
{
Order o = new Order(product, quantity, price);
_orders.Add(o);
}
//...
}
Now it looks better. At least from the have one reason to change principle. But I don’t see this improved testability at all. That is because although we split the responsibilities to separate methods, we left the methods tightly coupled to one another. Lets see what we can do about this.
public class Customer
{
public void PlaceOrder(Product product, int quantity)
{
CheckProductAvailability();
CheckCustomerCreditLimit();
CheckSomethingElse();
decimal price = CalculateTotalPrice(product, quantity);
Apply(new OrderPlacedEvent(product, quantity, price));
}
private void OnOrderPlaced(OrderPlacedEvent @event)
{
Order o = new Order(@event.Product, @event.Quantity, @event.Price);
_orders.Add(o);
}
//...
}
We introduced a concept of event as a way of encapsulating the state change we are about to apply. Now we can represent state changes by a series of events. The other thing we did was adding the Apply method. What is it? The Apply method is the API of Event Sourcing. It is a way of communicating hey, I, the aggregate root, want to apply the state change represented by this event! Apply invokes a method to apply state change. It can do this in a variety of ways: based on convention (like On… convention in this sample), an attribute or any other. The bottom line is, Apply applies the state change and it can be called externally, for example during tested state preparation.
[TestFixture]
public class CustomerTests
{
[Test]
public void When_placing_fifth_order()
{
Product product = new Product();
//Arrange
Customer c = new Customer();
c.Apply(new OrderPlacedEvent(product, 1, 10));
c.Apply(new OrderPlacedEvent(product, 1, 10));
c.Apply(new OrderPlacedEvent(product, 1, 10));
c.Apply(new OrderPlacedEvent(product, 1, 10));
//Act
c.PlaceOrder(product, 1);
//Assert...
}
}
You can see that we invoke no business logic during arrange phase. We only apply predefined, known to be right, state changes. The chance something bad will happen during this phase is minimal, as we certainly tested the OnOrderPlaced method in a different test, didn’t we?
This is what I call well tested code. PlaceOrder method is always tested in isolation, no matter if it is first or fifth order. One cannot overestimate the benefits of a testable codebase. If you happen to play (or work) with Domain-Driven Design, Event Sourcing can be beneficial for you. I see it as the missing piece of puzzle — the model is finally fully testable.
Ncqrs
May 30th
W ciągu ostatnich dwóch tygodni moją uwagę przykuł na dobre nowy framework open source — Ncqrs (witryna CodePlex Ncqrs znajduje się tutaj). Jak sugeruje nazwa, Ncqrs służy do budowy systemów w oparciu o wzorzec architektoniczny Command-Query Responsibility Separation (CQRS). To, czego nazwa nie mówi, to fakt, że Ncqrs narzuca pewną specyficzną implementację wspomnianego wzorca, a mianowicie tę opartą o technikę Event Sourcing. Na podstawie posta Grega Younga można by się czepiać, że nazwa Ncqrs nie jest zbyt trafna, ale odłóżmy kwestie nomenklatury na bok. Czym jest Ncqrs i jak to się stało, że mnie tak zafascynował?
Zasada działania
Ncqrs jest całościowym rozwiązaniem służącym do budowy systemów opartych o silny model domeny, którego stan jest przechowywany za pomocą strumienia zdarzeń. Poniższy diagram prezentuje workflow dla pojedynczego przypadku użycia w Ncqrs.
Komendy
Punktem wejścia do Ncqrs są komendy. Są to obiekty, który reprezentują żądania wykonania pewnej operacji na modelu domeny. Komendy są mapowane na operacje za pomocą rozbudowanego rozszerzalnego mechanizmu. Najprostsza implementacja mappera opiera się na dwóch atrybutach, które określają, czy dana komenda ma wykonywać metodę istniejącego obiektu (w nomenklaturze Ncqrs — korzenia agregatu), czy też tworzyć nowy obiekt. Properties komendy są mapowane (na podstawie nazwy) do parametrów wybranej metody lub konstruktora. Ostatecznie, (jeśli to konieczne) z magazynu danych podnoszony jest odpowiedni obiekt i wykonywana jest odpowiednia metoda (lub konstruktor).
Operacje biznesowe
Operacje i konstruktory obiektów są w Ncqrs wywoływane tylko za pośrednictwem komend. Ich jedynym zadaniem jest wykonanie logiki biznesowej. Nie mogą one bezpośrednio modyfikować stanu obiektu. Zamiast tego, dozwolonym mechanizm modyfikacji stanu jest zgłaszanie zdarzeń. Operacje biznesowe mogą także komunikować się ze światem zewnętrznym.
Stosowanie zdarzeń
Jak już wspomniałem na wstępie, zdarzenia są sposobem przechowywania stanu obiektów w Ncqrs. Operacja biznesowa może zgłosić jedno lub więcej zdarzeń. Dla każdego z nich framework wyszukuje odpowiedniej metody je przetwarzającej. Mechanizm ten jest oczywiście rozszerzalny, a out-of-the-box Ncqrs zapewnia dwa sposoby wiązania zdarzenia z metodą obiektu biznesowego przeznaczoną do jego obsługi: za pomocą konwencji oraz za pomocą atrybutów. Ncqrs wywołuje znalezioną metodę przekazując jej zgłoszone zdarzenie.
Przetwarzanie zdarzeń
Metoda przetwarzająca zdarzenie modyfikuje stan obiektu na podstawie danych przekazanych zdarzeniu. Tylko tyle i aż tyle. Metoda ta nie powinna zawierać żadnej logiki biznesowej (warunkowej), ani mieć jakichkolwiek skutków ubocznych (komunikacja z innymi systemami itp.).
Zapewne chcielibyście zapytać po co tyle komplikacji? Dlaczego operacja biznesowa nie może zmodyfikować stanu? Odpowiedź jest prosta. Ponieważ stan obiektów jest reprezentowany jako strumień zdarzeń przez nie wygenerowany, aby odtworzyć obiekt niezbędne jest stworzenie jego pustej instancji, a następnie przetworzenie (w kolejności!) wszystkich zapisanych zdarzeń — oczywiście za pomocą odpowiednich metod przetwarzających. Metody te są więc nie tylko stosowane do modyfikacji stanu podczas przetwarzania, ale także do odtwarzania tego stanu podczas podnoszenia obiektu z trwałego magazynu.
Dzięki takiemu podejściu systemy Event Sourcing (jak Ncqrs) zapewniają, za darmo, ślad audytowy, który ma gwarancję poprawności, ponieważ jest on wykorzystywany do budowy obiektów podczas normalnego działania systemu.
Publikowanie i denormalizacja zdarzeń
Jeśli wszystko do tej pory przebiegło prawidłowo, wszystkie zgłoszone zdarzenia są publikowane. Oczywiście, także w tym wypadku Ncqrs pozwala wymienić mechanizm publikacji zdarzeń. Domyślny wykorzystuje komunikację wewnątrz procesu, ale dostępny jest także taki, który wykorzystuje NServiceBus.
Publikowanie zdarzeń ma dwa cele. Po pierwsze, pozwala powiadomić zainteresowane systemy zewnętrzne o zmianach stanu naszego systemu. Polega to na eskalowaniu “lokalnych” (dotyczących naszego systemu) zdarzeń do statusu zdarzeń “globalnych” (mających znaczenia dla całego środowiska systemów). Stąd już tylko jeden krok do pełnej Event Driven Architecture (EDA).
Drugim celem publikowania zdarzeń jest tzw. denormalizacja, czyli aktualizacja podsystemu obsługi zapytań. Jaki podsystem? O co chodzi? Dokładny opis zagadnienia CQRS znajduej się tutaj. W tym miejscu wspomnę tylko, że systemy CQRS wykorzystują zwykle dwa osobne magazyny danych dla przetwarzania komend oraz do realizacji zapytań. Do synchronizacji tego drugiego magazynu danych wykorzystywane są właśnie denormalizatory zdarzeń. Proces denormalizacji polega na wykonaniu w bazie danych dla zapytań modyfikacji, które wynikają z opublikowanego zdarzenia. Skąd taka nazwa? Otóż zdarzenia stanowią znormalizowaną (pozbawioną redundancji) postać danych. W magazynie dla zapytań zaś, te same dane mogą mieć wiele reprezentacji, ponieważ nadrzędnym celem jest optymalizacja czasu realizacji zapytań.
Dlaczego to może działać?
Jest kilka powodów, które sprawiają, że (pozornie) szalona idea reprezentacji stanu obiektów jako ciągu zdarzeń może działać w praktyce. Oto kilka z nich:
- zapewnia darmowy, gwarantowany, ślad audytowy
- możliwość wykonywania tzw. snapshot’ów (czyli pełnych zrzutów zserializowanego obiektu biznesowego) co N zdarzeń. Dzięki temu odtwarzanie obiektu wymaga jedynie przetworzenia zdarzeń, które nastąpiły po ostatnim snapthot’cie
- w klasycznym DDD (z użyciem O/RM) podczas podnoszenia obiektu z bazy danych tak naprawdę pobierane jest wiele wierszy danych (np. za pomocą podzapytań i złączeń). W przypadku Event Sourcingu pobieranych jest kilka wierszy przechowujących zdarzenia. Oba podejścia mają więc podobną złożoność na poziomie bazy danych.
Zapraszam Was do zabawy z Ncqrs. Postaram się odpowiedzieć na wszystkie Wasze pytania.
