Writing an Event-Sourced CQRS Read Model

Discussions about event sourcing and CQRS seem to usually focus on the overall system architecture or various flavors of domain-driven design in CQRS context. However, the read models are often neglected, even though there are some interesting considerations on this side as well. In this post we’re going to present a sample implementation of populating a view model by consuming event stream.

Overview

The idea of a read model is really simple. You take the event log, apply (replay) all the events on an initially empty data model using appropriate functions, and you get the populated model. The code could look like:

List<Event> events = getEvents();
Model model = Model.empty();
for (Event event : events) {
    apply(model, event);
}

We can make this even shorter with functional programming:

Model m = reduce(getEvents(), Model.empty(), (m, e) -> apply(m, e));

That is the essence. Note that it is just the abstract outline and realistic implementation is likely to differ, including buffering, batching (or streaming), persistence etc.

Applying Events

The actual Java code to apply the events may look similar to the below:

EventProcessingResult processEvents() {
    if (getState().isRunning()) {
        int batchSize = getEventsPerIteration();
        List<Event> events = eventStore.getEventsForAllStreams(getLastEventId(), batchSize);
        if (events.isEmpty()) {
            return NO_EVENTS_TO_PROCESS;
        } else {
            return processEvents(events);
        }
    } else {
        return NOT_RUNNING;
    }
}

EventProcessingResult processEvents(List<Event> events) {
    try {
        for (Event event : events) {
            dispatchEvent(event);
        }
        return SUCCESS;
    } catch (RuntimeException e) {
        return FAILURE;
    }
}

All in all it’s really simple and straightforward. It is possible to enhance it with hooks before and after processing individual events and the entire batch. Such hooks could be used to:

  • implement transactions,
  • plug in monitoring,
  • implement error handling,
  • calculate the batch size depending on speed,
  • perform arbitrary operations, e.g. setting something up or recalculating once per batch.

The last interesting piece is the dispatchEvent method. Aside from walking the type hierarchy, error handling and making it all optional, it boils down to:

void dispatchEvent(Event e) {
    Method handler = projector.getClass().findMethod("on", e.getClass());
    handler.invoke(projector, e);
}

In other words, for each event type (like OrderCreated), we look for a public method called on that takes a single argument of matching type, on a projector object.

All of the above is part of an engine, a piece of infrastructure backing many view models. All that is necessary to implement a projection is actually provide the projector, with handlers for interesting event types. All other events will simply be ignored.

It could look like this:

public class OrderProjector {
    @Inject
    private OrderDao orders;

    public void on(OrderCreated e) {
        orders.save(new Order(e.getOrderNumber()));
    }

    public void on(OrderApproved e) {
        Order o = orders.find(e.getOrderNumber());
        o.setApproved(true);
    }
}

Projection Thread

Let’s discuss multi-threading for a moment. Shared mutable state immediately brings numerous problems and should be avoided as much as possible. One of the ways to deal with it is not having concurrency in the first place, e.g. by limiting writes to a single thread. In most cases a single-threaded writer combined with ACID transactions is more than enough to keep up with the write load. (The read/query load can be heavy and use many threads – all of the details here are only about the writes.)

The thread is responsible for applying the events to the read model, all the way from querying the event store to updating the view model database. Normally it just loads batches of events from the store and applies them. It continues as long as there are more events to process, and goes to sleep after it’s caught up. It wakes up after a certain amount of time or when notified about new events by the event store.

We also have some control over this thread’s life cycle. For example, we have a way to programmatically pause and resume each projection’s thread, even exposed in an admin GUI.

Push or Pull?

With a database-backed event store, it’s very easy to query repeatedly for new events. This is the pull model. Unfortunately, it also means that you may end up polling too often and generating needless load, or polling too infrequently and thus possibly taking longer to propagate changes to the view model.

That’s why in addition to polling the event store it’s a good idea to introduce notifications that wake up the read models as soon as new events are saved. This effectively becomes a push model with minimal delays and load. We found JGroups to be a very good tool for the job – it supports multiple protocols and is very easy to set up, involving much less hassle than a full-blown message queue.

The notifications may or may not contain actual events.

In the latter (and simpler) design, they only spread the information that a new event has been saved, along with its sequential ID (so that all projections can estimate how much behind they are). When awakened, the executor can continue along its normal path, starting with querying the event store.

Why? Because handling events coming from a single source is easier, but more importantly because a DB-backed event store trivially guarantees ordering and has no issues with lost or duplicate messages. Querying the database is very fast, given that we’re reading a single table sequentially by primary key, and most of the time the data is in RAM cache anyway. The bottleneck is in the projection thread updating its read model database.

However, there are no obstacles to putting event data in the notifications (except for maybe size or network traffic considerations). It would likely decrease the load on the event store and save some round trips to database. The projector would need to maintain a buffer and fall back to querying the event store when needed. Or the system could use a more reliable message queue.

Restarting Projections

Aside from pause/resume, the above screenshot shows one more action: restart. Innocuous as it looks, it’s a really nice and powerful feature.

Since the view model is completely derived from the event log, at any time it can be thrown away and recreated from the beginning (or from some initial state/old enough snapshot). Data is safe in the event log, the ultimate source of truth.

It’s useful when anything about the view changes: a field or a table is added, a bug is fixed, something is calculated differently. When it happens, it’s often easier (or required) to just start from the beginning, rather than for example implement massive SQL migration script.

It’s even possible to go as far as fully automating it, so that when the system starts up and it detects the DB schema does not match the corresponding Java model, it can automatically recreate the schema and reprocess the event log. It’s like running with Hibernate create-drop policy, except for that it doesn’t lose data.

Performance

The solution may appear quite limited with regards to performance. One point that could raise an eyebrow is the single-threaded writer. In reality a single thread is usually fast enough to easily keep up with the load. Concurrency is not only more difficult to implement and maintain, but it also introduces contention. Reads (queries) can be heavily multi-threaded and easy to scale out.

We also gain a lot by having multiple read models, for example separating analytics from administration and “transactional” data. Each model is single-threaded (for writing), but the multiple models consume events in parallel. Finally, the solution could be modified to use sharding or some kind of fork-join processing.

Another interesting point is restarting projections from scratch.

A good solution is something like kappa architecture:

  • Keep the outdated projection up and running and answering all the queries.
  • Start a new projection, e.g. to another database. Just let it process the events, don’t point any traffic to it.
  • When the new projection catches up, redirect traffic and shut down the old one.

On a very small instance, especially for development, it may even be possible to do a restart online, on the same instance. It depends on answers to the following questions: How long does it take to reprocess all events? Is it acceptable for this projection to be stale for 30 minutes? Can we deploy at night or weekend, when nobody is using the system anyway? Do we have to replay all the history?

Another factor to consider here is persistence. If it’s too much of a bottleneck and cannot be further optimized, consider using in-memory view models.

Summing Up

In essence, that’s all it takes to implement a read model consuming an event store. It gains much simplicity thanks to a linear event store and processing everything in a single thread. So much that in the end it’s really just a loop, implementing the reduction shown in the beginning.

In future posts I am going to dig deeper into practical concerns of implementing projections.

 

Angular 2 Plans – Angular Boot Camp

Here at Oasis Digital, we have been receiving lots of questions about our plans around our popular Angular Boot Camp class, and the upcoming Angular version 2. It is important to handle transitions like this well; here is our plan, already underway.

Angular 2 is Coming Soon!

We are already hard at work with Angular 2, and have been for some time. We have an Early Start with Angular 2.0 class already scheduled – first in St. Louis (almost sold out!) then in San Francisco – see the page for details.

How soon is Angular 2 coming? There isn’t an official answer yet (as of September 2015), but we are hoping for an announcement at the major European Angular conference Angular Connect in October – By the way, we are a sponsor, and are offering the Angular Boot Camp a few weeks later near London.

We are further hoping, but have no evidence other than reading the tea leaves, that version 2 will ship at or around ng-conf 2016. Our plans are roughly based upon this assessment – we will adjust right away as further information about the schedule emerges.

From a technical readiness point of view, we have been watching Angular 2 quite closely, and feel like it is now converging on a ready-to-use state, after extended early development. Keep an eye on this blog for our Angular 2 content.

AngularJS 1.x Training

We will continue to support our AngularJS 1.x customers after Angular 2 ships. We expect AngularJS 1.x will continue to be under heavy use at many organizations for years to come. We plan to continue offering a AngularJS 1.x-centric class for as long as customers need it.

Our Angular Boot Camp already includes content about getting ready for version 2, and we will continue to refine and increase that based on the stream of alpha, beta, and then release versions of Angular 2.

Angular 2.x Training

As mentioned above, you can sign up now for our Early Start with Angular 2.0 class. This class is intended for developers already familiar with AngularJS 1.x, and primed to move quickly to new suite of technologies for 2.0.

At some point, as the demand develops, we will offer our full 3-day “boot camp” experience in an Angular 2.0 class. It will provide everything a new developer needs to get up and running with the new version, with some attention paid to compatibility with the old version, but taught in a way that works regardless of whether developers are coming from AngularJS 1.x.

Another interesting question is how to handle customers who realistically need to work heavily in both. We may offer a combined class, four or five days, teaching both – we are not sure if this is ideal yet, it will depend on our customers’ needs.

Update: We now offer a full 3-day Angular 2 class, starting in 2016.

 

Getting Started With DI in Angular 2

One of the core features of Angular 2 is dependency injection. This post serves as an introduction to using DI in Angular 2 apps.

Hierarchical DI

Angular 2 uses a hierarchical system for managing injectable objects. For developers this means that the choice of having a new instance or a previously created one is up to the consumer of the injectable. Let’s see how this works. Consider the following example of a single component and a single service (service definition not shown):

@Component({
    selector: 'parent-component'
    bindings: [MyUtilsService]
})
@View({
    template: '<h1>Hello World</h1>'
})
class ParentComponent{
    constructor(myUtils: MyUtilsService) {
        console.log(myUtils.getMessage());
    }
}

In the above sample line 3 creates a binding to “MyUtilsService”. This informs Angular that ParentComponent needs a fresh injector to provide access to MyUtilsService. The reference to MyUtilsService on line 9 is the actual request to perform injection. At this point Angular will create a new instance of MyUtilsService which is available in ParentComponent’s constructor. Let’s add another piece.

@Component({
    selector: 'child-component-1',
})
@View({
    template: '<h2>Child 1 Content</h2>'
})
class ChildComponent1{
    constructor(myUtils: MyUtilsService) {
        //Same instance of myUtils as injected into ParentComponent
        console.log(myUtils.getMessage());
    }
}

@Component({
    selector: 'parent-component'
    bindings: [MyUtilsService]
})
@View({
    template: '<child-component-1></child-component-1>'
    directives: [ChildComponent1]
})
class ParentComponent{
    constructor(myUtils: MyUtilsService) {
        console.log(myUtils.getMessage());
    }
}

In this example ChildComponent1 is also referencing an injection of MyUtilsService. Since a binding was not specified for MyUtilsService on ChildComponent1 Angular injects the same instance that was created for the ParentComponent. But what if we wanted a new instance? Let’s say that ParentComponent manipulates the state of MyUtilsService in some way that makes its use undesirable in some additional child component. To achieve this a new instance would be needed instead of using the parent’s copy.

//ChildComponent1
@Component({
    ...
})

//ChildComponent2
@Component({
    selector: 'child-component-2'
    bindings: [MyUtilsService]
})
@View({
    template: '<h2>Child 2 Content</h2>'
})
class ChildComponent2{
    constructor(myUtils: MyUtilsService) {
        //This references a different instance of myUtils from ParentComponent and ChildComponent1
        console.log(myUtils.getMessage());
    }
}
@Component({
    selector: 'parent-component'
    bindings: [MyUtilsService]
})
@View({
    template: '<child-component-1></child-component-1><child-component-2></child-component-2>'
    directives: [ChildComponent1, ChildComponent2]
})
class ParentComponent{
    constructor(myUtils: MyUtilsService) {
        console.log(myUtils.getMessage());
    }
}

Now a new instance of the MyUtilsService is requested on line 9 by creating a child injector. As a result, ChildComponent2’s constructor will be supplied with a different instance than was injected into the ParentComponent. Lets add one more layer and assume that ChildComponent2 is the only component in this hierarchy in need of a separate instance of MyUtilsService. ChildComponent2’s child wants to inject the same instance that was created for and injected into ParentComponent. One may assume the following approach can be taken:

//ChildComponent1
@Component({
    ...
})

//ChildComponent2
@Component({
    selector: 'child-component-2'
    bindings: [MyUtilsService]
})
@View({
    template: '<grandchild-component></grandchild-component>'
    directives: [GrandchildComponent]
}

//GrandchildComponent
@Component({
    selector: 'grandchild-component'
    bindings: [MyUtilsService]
})
@View({
    template: '<h3>Grandchild Content</h3>'
})
class GrandchildComponent{
    constructor(myUtils: MyUtilsService) {
        //This will look for and find the instance of MyUtilsService created by ChildComponent2 
        console.log(myUtils.getMessage());
    }
}

//ParentComponent
@Component({
    ...
}

However, the GrandChildComponent is receiving a third instance of MyUtilsService. This is also not ideal. To correct this and achieve the desired result let’s revisit ChildComponent2 and make a few small adjustments.

//ChildComponent1
@Component({
    ...
})

//ChildComponent2
@Component({
    selector: 'child-component-2'
    viewBindings: [MyUtilsService]
})
@View({
    template: '<grandchild-component></grandchild-component>'
    directives: [GrandchildComponent]
}

//GrandchildComponent
@Component({
    selector: 'grandchild-component'
})
@View({
    template: '<h3>Grandchild Content</h3>'
})
class GrandchildComponent{
    constructor(myUtils: MyUtilsService) {
        //This will look for and find the instance of MyUtilsService created by ChildComponent2 
        console.log(myUtils.getMessage());
    }
}

//ParentComponent
@Component({
    ...
}

View bindings allow for the creation of injectors that will create instances of an injectable that are only available to the component they are declared on. When MyUtilsService is injected into the GrandchildComponent the DI system will climb the hierarchy looking for an injector that is aware of MyUtilsService. The instance created by ChildComponent2 is not present in the hierarchy and instead the instance created on the ParentComponent is used.

App Level Instances

In addition to creating dependency bindings at the component level it is also possible to specify a set of top level, application dependencies. This is done using the bootstrap function to initialize the application. For example:

bootstrap(ParentComponent, [MyUtilsService])

Now any component of the application can inject the same instance of MyUtilsService or request a new copy using the binding approaches listed above.

Conclusion

There is great flexibility and power in the new dependency system. This article covers some of the basics that can be used to get your application off the ground but there is still plenty more to learn about how it works under the hood. For more information about how the Angular 2 dependency injection system works check out one of the following resources:
http://victorsavkin.com/post/102965317996/angular-2-bits-unified-dependency-injection
https://angular.io/docs/js/latest/api/di/

Angular Boot Camp, London UK

A number of companies in Europe have inquired about our Angular Boot Camp immersive Angular class, and a few threatened to send students across the ocean to attend one of our public classes. So when we saw that the 2015 European Angular conference (Angular Connect) is in easily-accessible London, we stepped in as a sponsor, and decided to offer Angular Boot Camp a few weeks thereafter, also in London.

After many weeks of paperwork delays, we have been issued the appropriate VAT number to do business there, and are pleased to announce our first offering of Angular Boot Camp across an ocean. The venue is a training facility in Wokingham, not far from Heathrow and easily accessible by train to downtown London. The dates are 3-5 November 2015, please see the site for more information and to register.

 

Achieving Consistency in CQRS with Linear Event Store

In a recent project involving an event-sourced CQRS system, we decided to do some things that seem somewhat unusual compared to solutions mostly talked about. However, they let us achieve some nice properties that would be hard (if possible at all) otherwise.

Event Store as Regular Table

We decided to implement the event store as a regular table in an RDBMS. We used PostgreSQL, but there is little PostgreSQL-specific here. We know this database is very reliable, powerful and simply mature. On top of that, single-node ACID transactions provide some really nice benefits.

The table ended up with the following fields:

  • event_id (int) – primary key coming from a global sequence
  • stream_id (UUID) – ID of an event stream, typically a DDD aggregate
  • seq_no (int) – sequence number in history of a particular stream
  • transaction_time (timestamp) – transaction start time, the same for all events committed in one transaction
  • correlation_id (UUID)
  • payload (JSON)

Not all of them are mandatory for an event store, but there is one important and uncommon difference: event_id – globally, sequentially increasing number. We’ll get to that in a moment.

Can You Do It?

If you go for an event store in a regular DB table, getting a global event ID like this is extremely cheap. Databases are really efficient generating, storing, indexing etc. such columns. The only actual problem is whether you can afford using a DB table for it in the first place.

The limits for an append-only relational table are much higher than most applications will ever need. Postgres can easily handle thousands (or tens of thousands) of writes per second.

Using a relational database certainly adds overhead and it’s not something I’d recommend if you were building the next Amazon. But chances are you aren’t, and so you may be able to afford the luxury of using simpler technology.

Benefits of Global, Sequential Event ID

Now that we have this peculiar event ID, what can we do with it?

Let’s have a look at the read interface of our event store:

public interface EventStoreReader {
    List<Event> getEventsForStream(UUID streamId, long afterSequence, int limit);
    List<Event> getEventsForAllStreams(long afterEventId, int limit);
    Optional<Long> getLastEventId();
}

The first method is pretty obvious and something you can find everywhere. We only use it to restore a single stream (aggregate) from the event store for handling a new command.

The other two are using the event ID, returning a batch of events after a particular event, and ID of the last event. They are the base of our read models (projections).

Read models are implemented by polling (with hints) the event store. They remember the ID of the last processed event. Every once in a while (or when awoken by a notification from the event store), they read the next batch of events from the store and process them in sequence, in a single thread.

This kind of linear, single-threaded processing is probably as simple as it can get, but it obviously has limited scalability. If you get 600 events per minute, it means on average you cannot be slower than 100 ms per event, no matter what. In reality you also need to consider overhead and leave some headroom, so it needs to be faster than that.

It can be addressed with sharding or parallelizing writes in the read model, but for the moment we did not find it necessary. Having multiple independent, specialized models running in parallel certainly helps with that.

Comparing the last-processed event ID for a projection to the current global maximum, you can immediately tell how much behind the projection is. It’s the logical equivalent of queue size.

The global sequence can also be used to mitigate the downsides of eventual consistency (or staleness).

Executing a command could return the ID of the last written event. Then a query can use this ID, requesting: “I’m fine waiting 5 seconds, but don’t give me the result if your data is older than this ID”. Most of the time it’s a matter of mere milliseconds. For that price, when a user makes a change, she immediately sees the results. And it’s the actual data coming from the server, not simulation achieved by duplicating domain logic in the user interface!

It’s also useful on the domain side. We have some application and domain services that query some domain-specific projections (e.g. for unique checks). If you know that the last event in the event store is X, you can just wait until the projection catches up to that point before making further progress on the command. That’s all it takes to address many problems typically solved with a saga.

Last but not least, since all events are ordered, the projection is always consistent. It may be behind by a few seconds, or a few days, but it’s never inconsistent. It’s simply impossible to run into issues like having one stream processed until Monday, but another until Thursday. If something had happened before a particular event occurred, the same order is always maintained in the view model.

It makes the code and the state of the system a lot easier to write, maintain, and reason about.

Scalability and Complexity Revisited

There is a tendency to use complex, high-scalability technology regardless of the actual customer requirements and realistic scale. Such tools have their place, but they’re not obvious winners, no golden hammers solving all problems. Moreover, they are really expensive, if you consider the complexity of development and operations, and their limits.

Sometimes a simpler tool will solve the problem well. Not only do you save on development and ops, but also gain access to some really powerful tools that are impossible at high scale. Including global counters, linearizability and ACID transactions.

SQL databases are really fast and come with many useful tools in the box. Linear processing aligns well with hardware limitations, and its limits in general (beyond SQL) are at least in the order of millions of transactions per second for a single thread.

There are many good reasons to choose boring technology. If you innovate (and you should), be careful with why you actually do it, and don’t innovate in all areas at the same time.

Early Start with Angular 2 – New Class Offering

We have been using Angular version 1.x for years now, and following version 2 development with great interest. Things have come together greatly over the last few months, after having somewhat wandered prior to that. We feel like now is the time to dig into version 2.

There been comments by the core Angular team also, which indicate that now is the time for developers to start previewing the new release.

For companies that want to dig in deep, we are now offering a brand-new “Early Start with Angular 2” class. The first scheduled date is in November in St. Louis, the second in early December in San Francisco.

(This does not replace our Angular Boot Camp class, it is firmly oriented toward immediate production use of Angular, and will remain so at least until version 2 ships as a final production form, and perhaps even thereafter for companies still using version 1.)

 

 

Introduction to Event Sourcing and Command-Query Responsibility Segregation

The concepts of event sourcing (ES) and command-query responsibility segregation (CQRS) have been around for quite a while. They are getting more and more attention in the Java community, though they seem to have been much more popular over on the .NET side.

I have spent the last few months implementing a system with such an architecture at Oasis Digital. While working on it, we aimed for flexibility and the ability to make the right trade-offs and choose our own tools. Some interesting solutions came up in the process, things that would probably be impossible with existing frameworks like Axon.

I am going to write more about some of these pieces in the future. Before we get there, let’s start with a brief introduction to event sourcing and command-query responsibility segregation.

CQS

The main idea of command-query separation (CQS) is that all operations are either:

  • commands, changing state of the system,
  • queries, getting some information from the system.

Either one or the other, never both. For example, if a command changes anything in the system, it should not be used to read its state. Mutation-free read access should always be possible. Asking a system to change something to read a value seems plain wrong, and queries inadvertently changing state are very confusing, surprising and leading to bugs at best.

These principles can be applied on all levels. Uncle Bob in “Clean Code” calls it one of the main principles of good function design. The pattern is also applicable to system design.

CQRS in System Architecture

Command-query responsibility segregation (CQRS) is a pattern in system architecture inspired by CQS. It divides the system in two distinct parts, separating the components used for writing (executing commands) from those for querying. In such a system we can find two kinds of requests corresponding to these two models.

AC stands for Autonomous Component

Firstly, there are commands – ordering the system do something or change its state. A piece of business logic updates the domain model (or rejects the command) and lets the client know that the change has been accepted (or not).

The business logic is often implemented using domain-driven design, but it may just as well be transaction script or any other applicable technique. Actually, we ended up using a mix of these two in one area of the system.

Whenever anything changes, the domain model somehow notifies everyone – for example by publishing domain events. These events are received by view models, which update their own representation of the state, separate from the domain model.

This leads us to the second kind of requests: queries, getting information from the system without changing its state. These only use the view models, not the domain.

What is especially powerful about this pattern is the separation of concerns.

The domain (write) side is all about the business. It does not care about queries, all the different ways to show this information to the users, performance, optimal storage for these purposes and all that.

On the other hand, the query (read) side is all about the read access. Its main purpose is making queries fast and convenient.

While the domain is only implemented once, there can be multiple view models over the same data. They often use different databases. They can even be using different technologies – all kinds of NoSQL, normalized and denormalized SQL, in-memory representations, Lucene indexes, OLAP cubes etc.

The read models are also free to use different languages, if they make anything easier. There are no obstacles to implementing the domain model in Scala, but doing view models in Clojure or even SQL stored procedures.

Overall, the view models are very good (and safe) candidates for innovation.

Unlike the domain model, code quality in view models does not have to be perfect. They’re all about reshaping data and moving it around to make reads convenient. Some shortcuts and dirty tricks may be acceptable, as long as they don’t make the whole thing unmaintainable.

Read models are often denormalized, prepared to answer concrete questions in optimal way. It can even be as extreme as having a precomputed set answers to every query that the system will handle, stored in some trivial key-value way.

We often call the view models “projections” – because they “project” the domain events to a particular model, keeping only as much information as is necessary and in optimal shape for queries they serve.

Note that all the domain logic is only implemented in the domain (write) model. It’s done once and in the right place. Even if a value is directly derived (calculated), as soon as it has a business meaning or is calculated with some business logic, it belongs in the domain model.

Event Sourcing

Another pattern commonly used with CQRS is event sourcing.

In short, it means that all data in the system is stored in the form of events, in an event log. An event is a piece of information stating that something has occurred (user created, name changed, shipping address added, order submitted, order delivered). They’re always in past tense, saying that something has happened.

The events never change. You can never delete or update them. If something has happened, it’s happened. If it was a mistake, it can be corrected with a complementary action generating a new “inverse” event, but there’s no going back and saying it has not happened.

Combining ES and CQRS

Event sourcing and command-query responsibility segregation perfectly fit each other. It’s a powerful synergy effect: each of them becomes more powerful thanks to the combination.

When a command comes in, the domain model calculates the new state of the system and possibly emits some new events (which are the only way the changes are persisted). When another command comes in for the same logical area, the domain model is restored from the past events, and responds to the new command by generating some new events. The events represent concrete changes that have business meanings. Technically in a sense they are “deltas” (or “diffs”) of the system state.

The view models simply handle these events, picking up only these they are interested in, and updating their state to support future queries.

Benefits

There are many benefits to using this approach, let me just point out a few of them.

CQRS naturally leads to task-based UIs. Every action represents some very concrete business event. There can be a huge difference between changing someone’s name, changing their shipping address, or making them a gold customer. If end user wants to change name, they get a small form that has exactly what is needed for that operation. If they make a customer gold – that’s another action in another form.

Contrast this with the traditional CRUD, spreadsheet-like systems. They have no such operations as “change name” or “change customer status to gold”. All the users can do is “change user”. Implementing logic that changes something when a particular field changes all of a sudden becomes harder. Validation is a nightmare. Auditing and simply seeing when something happened and what the users did, is impossible without adding more layers of complexity on top.

It is harder for the users, too – the use cases have to be implemented in their minds, knowing what to change when to achieve a desired effect.

Related to the above, and a reason why CQRS is often used with domain-driven design, is that the shape of the system naturally maps to business contexts and use cases. Commands correspond to concrete user intents, and queries are designed to answer concrete questions. Once again, it’s the exact opposite of a glorified spreadsheet-like DB frontend.

It’s applicable on higher level, too: Different areas of the business (bounded contexts in the DDD lingo) can be implemented as separate models. For example, a warehousing context can represent a book in a completely different way than a sales context. One may be interested in its size, weight and number on stock. The other – in author, genre, cover image, publisher description etc.

Event sourcing also makes reporting a lot easier. By definition it never loses information. Maybe yesterday you did not need to know how often users add and remove items from the shopping cart, and only cared about the orders they eventually submitted. But today business wants to trace this information, so maybe they can discover items that clients wanted but changed their mind. It may be worthwhile to tempt them with these items in future ads.

Answering such a change in a “traditional” system would be a nontrivial effort. With ES+CQRS chances are that all it will take is just another straightforward projection of data that is already there, and immediately answer questions about the past!

Finally, another obvious benefit is performance. Since the view models are separate, they can have very different schema. Avoid joins, keep data denormalized, answer many questions in linear or even constant time. Read-only access is much easier to scale, as is the write side which doesn’t care about queries anymore.

Costs

ES+CQRS is not without cost, and is not the best approach to all systems. More about this in a future post.

More Resources

Like I said in the beginning, these ideas have been around for quite a few years. There is a huge number of resources available online, in books and at conferences. This post has merely scratched the surface and is only meant as (yet another) humble introduction to the topic.

Here’s a few links to the masters:

Learn TypeScript Now, Prepare for Angular 2.0

Like most organizations building client-side web applications (SPAs), here at Oasis Digital we have most commonly written code in plain old JavaScript, i.e. ES5, with help from linters, unit tests, etc. Lots and lots of code is out there, written in ES5 plus AngularJS and other frameworks.

There are obvious weaknesses to this approach, or rather gains that are set aside by not embracing any of the numerous newer innovations. We have dallied with CoffeeScript, worked on minor projects with ClojureScript, and so on. There are a lot of great languages out there beyond the JavaScript, but statistically JavaScript/ES5 is the winner.

(For a bit of “inside baseball”: we at Oasis Digital are generally quite willing to use unpopular, better tools. So why ES5? Because of our secondary line of business, training. When trying to train someone to use a new JavaScript library or framework, the most likely common ground we will have with them is baseline ES5.)

However, the era of mostly writing JavaScript code as it will execute in the browser is ending rapidly. Innovation of the JavaScript language is running far ahead of what browser vendors will keep up with; and even to the extent browser vendors keep up, large projects will inevitably have to support today’s browser versions (with the JavaScript version embedded therein) for years to come. A build process with compilation to JavaScript will be part of most SPA projects in the future.

So the question is, which post-JavaScript language to pick up? My personal favorite is ClojureScript, though I also see much merit in type-inference-equipped functional languages like Elm. For mass use though, I suspect the answer is simply a JavaScript successor. This is where it gets tricky. ES2015 (wow, I miss the shorter “ES6”) is the standard, and it will eventually land in all the major browsers. It will “win” in terms of mass adoption. Today is a good day to start moving your server side (NodeJS) code, test code, and as much as possible all of your JavaScript development to ES2015 or even a bit beyond – we’ve already done so. Babel works very well. Traceur works very well.

typescript2But there is at least one major exception. Angular 2.0 (which is coming soon, who knows how soon?) is built in TypeScript. You can go look right now, the source code in progress is sitting there in GitHub, and it consists mostly of TypeScript files. Most of the examples out there (including those we have published) use TypeScript. It will certainly be possible to write Angular 2 applications using plain JavaScript or any other language that compiles to JavaScript, but the first class citizen will be TypeScript.

Therefore, if you are using Angular today, and want to upgrade to Angular 2 someday, start learning TypeScript. Start using TypeScript. Start using TypeScript in some of your AngularJS 1.x work (we do).

Even aside from the obvious Angular motivation above, type systems in general (and TypeScript in particular) can be very useful. They enable a much more helpful editing/IDE experience. Numerous bugs can be caught at compile time. The incidence of runtime errors tends to be much lower. Unit testing can carry less of the load when using a typed language, so that whatever amount of effort you devote to unit testing, a greater share of it can go toward important functionality versus more trivial tests.

Here are resources to get started.

We will follow up later with examples of moving Angular 1.x code to TypeScript; we might even add a class offering to teach TypeScript for AngularJS developers. Stay tuned.

Angular 2: Working With Alphas

At the St. Louis Angular Lunch this month, I talked about working with the recent Angular 2 alpha versions. The rapid changes to the Angular framework have made it difficult to pin down its usage. As a result information explaining how one might build a full application has been mostly non-existent. The goal of this talk was to explain some of the big picture concepts of how Angular 2 applications are composed and provide live coded examples.  See the video embedded here, and transcript below.

 

Transcript

Here is the talk, transcribed to text. This is a lightly edited, draft transcription, so any errors are probably from that process.

Continue reading Angular 2: Working With Alphas

Rapid Prototyping

Here at Oasis Digital we work on many kinds of projects. We start new projects in a “green field”. We revive old projects where former developers are no longer around. We teach classes open to the public, we travel to our customers to teach classes privately. We attack entire projects. We supply brainpower to existing projects.

Among the most fun work: to rapidly create a working-prototype implementation of a new idea.

In projects like this, it is often important to deliver quickly. To make that possible, the right goals and constraints are important. I have listed some of the possible goals and constraints below. Some of these conflict with each other! This is not a checklist of things to always do, but rather a list of ideas to consider.

  • Reduce the scope of the working-prototype as far as feasible, but no further.
  • Choose a team small enough to minimize coordination, but maximum parallel productivity.
  • Choose a programming language and other tools we know best.
  • Choose a different language and/or toolset, more well-suited to creating maximum functionality with little code for the problem at hand – even if it is not among those where we have the most experience.
  • Support only the most important browser or two, omitting legacy browser support.
  • Implement a wide swath of features, but very shallowly.
  • Choose the most vital/informative parts of the intended software system, and implement those in a working prototype manner, leaving the less interesting parts as static mockups.
  • Start with a very plain visual appearance, to make it obvious it is “just” a prototype.
  • Start with a more polished visual appearance, perhaps using an off-the-shelf template, to show the level of polish that will be possible in a more complete project.
  • Even if the system may eventually be a complex, multi-tier widely distributed system, implement the prototype as client-side only system; no server-side data storage at all. Often enough data can be stored in “HTML 5 local storage” to make it clear whether the project is actually a good idea and should proceed past a mere prototype.
  • Implement the prototype using something that approximates (but simplifies) the intended long-term architecture, because the prototype may need to demonstrate that it is a suitable architecture.

The final appealing thing about rapid prototype projects: they are straightforward to propose and price, often with a simple statement of work and fixed/firm price.

St. Louis Atlassian User Group – Aug 2015

On Aug 11, 2015 the St. Louis Atlassian User Group (AUG) met at the Urban Chestnut Brewery in the Grove area of St. Louis. This meeting was set up as a fairly informal gathering to share our experiences and make connections. It was a very successful event attended by nearly 70 people. The St. Louis group has been growing very rapidly this last few months.

We (Oasis Digital) made a video about the event:

We also participated in the professional side of the event. I led the discussion at the JIRA table, where people could bring their current challenges to get input and share ideas. There were also two other tables related to Confluence and a specific plugin from Siebert Media.

In addition we were also celebrating an Atlassian milestone, 50,000 organizations now use Atlassian products. This is up from 40,000 at the last JIRA Summit last October, very impressive growth. We are proud to be associated with Atlassian through our JIRA Boot Camp class.

Preparing for Angular 2.0

Here at Oasis Digital, we have been excited about the possibilities for Angular 2.0 ever since its (somewhat contentious) public debut at the Fall 2014 European Angular conference. ripThat was the conference with the infamous “tombstone” slides, and the frequently misunderstood explanation by the Angular team members that they did not have a migration strategy for Angular 1-to-2 yet. Many commenters at the time seemed to misunderstand the yet part as some assertion that there would be no migration strategy at all. Fortunately, the agitation around that initial announcement has subdued, work has continued, progress is being made.

Though there has been quite a lot of progress on Angular 2, which we have been following (much of the development is being conducted in the open) our recommendation now is that if you don’t have a reason to work with A2 well before its release, you’re probably better off ignoring it a bit longer. As I write this in August 2015, the Angular 2 alpha versions are still rather rough, particularly in the experience of getting started.

But at Oasis Digital we do have a compelling reason to start using new versions before they are ready. We must do this so that we have significant experience when the time comes to start teaching our students (for our Angular class) how to use the new version, and even for the teaching we already include in the class about how to prepare for Angular 2.

One of our primary trainers, Paul Spears, has been leading the charge. Here are links to a couple of repositories he has published, showing a minimal project to get up and running with Angular 2, loaded using JSPM:

There is quite a lot to explain about what is going on in those repositories, here are a few of the most important bits.

We expect that many teams (perhaps most) will concurrently upgrade their development/build tools to the current generation of ES6/2015 module-centric build systems. The leader in the space appears to be JSPM, so we have chosen that for this work. It is also possible (and we will likely publish example this way in the future) to simply load Angular 2 along with its required dependencies using the script tags from downloaded files or from a CDN, of course.

The JSPM configuration may look quite alien compared to the familiarities of Grunt or Gulp. This is because of the above-mentioned module loader approach, and because JSPM already understands many things that must be explained to Grunt or Gulp by the user or a plugin.

These examples are set up for development, loading numerous separate tiny libraries using numerous HTTP requests; but JSPM and similar tools also have an excellent solution for production asset preparation, using essentially the same configuration information to package all of the JavaScript into a file or two.

These repositories show working examples as of right now, early August 2015. Angular 2 has been changing quite substantially from one alpha to the next. Moving up to the next version will likely require adjustments, as it has in previous versions. This is definitely alpha software.

The config.js file looks quite scary, but much of the contents there are generated by JSPM tooling. With some adjustment, it is possible to omit the hairiest block of information therein from source control, and instead treat it as a build artifact. As you look at the contents of that file, remember that you will not have to write the bulk of it.