Merge Queue Tools

Introduction

In small projects, there are typically a few developers and a few branches underway. As those branches reach readiness, depending on the flow of the project each developer may merge their own branches, or a tech lead may merge on request. Or sometimes they pile up (in a queue, conceptually) for review and merge.

But in large teams working (either a large single project or a monorepo) there can be dozens/hundreds/thousands of developers creating branches all the time. Many teams have the practice of doing a modest or moderate amount of work per branch, resulting in a large number of branches that must be merged to move the work forward.

To be merged, a change (branch) needs to be at least conceptually on top of the current mainline (regardless of whether a merge-commit or rebase strategy is used), and tests/lint/build/etc must pass. As each merge happens, the baseline moves forward – so the next candidate sometimes can’t be accurately tested until that point. At scale, this becomes the limiting factor in system progress: If the tests take 1 hour, the maximum number of merges you can do per day can be as low as 24.

Long before that limit though, having humans perform each of these operations is impractical, so teams typically adopt an automated system to queue up proposed changes/branches for merge. The function is roughly:

  1. Developers submit a branch as ready for merge, into a queue; often mediated by a human review and approval process.
  2. Automated merge system keeps picking the top thing in the queue.
  3. Rebase it on the current mainline.
  4. Test, Lint, Build, etc.
  5. Hopefully, merge.
  6. Repeat.
  7. When a change fails to re-baseline / merge / test, notify the developers working on that branch, they will need to resolve either a syntactic or semantic merge conflict and resubmit.

Of course there are strategies to increase parallelism; the process remains conceptually serialized but practically parallelized.

  • Related or unrelated branches can be combined and tested + merged as a group.
  • Use a build system that has a great understanding of what subset must be re-tested for a given set of changes.
  • Pre-build each candidate on a recent baseline, to pre-populate a build cache; a well-crafted build process (with Bazel or other tools) can safely optimize away some of the rebuild-everything process.

Some Tools

Here are some of the available merge queue tools – but many organizations create an internal merge queue management system.

For now, use of an automated merge queue remains mostly a marker of large-team big-tech-co needs; over time I expect it to be a routine in-the-box tool almost as common as CI.

Thinking hard about a project launch

Here at Oasis Digital, we are always agile, and depending on the project needs, sometimes use Agile (in the “capital A” sense) processes extensively. Yet regardless of agility, iterations, steering, and so on, the planning and decisions made at the beginning (and in the early months) of a project often have profound and very difficult to change consequences later.

Therefore, while we would never argue for the straw-man Waterfall, we aim to think very hard about a project at the beginning.

Continue reading Thinking hard about a project launch

Querying without OR in Firestore

Background

Here at Oasis Digital we have successfully used the Firebase Realtime Database, and more recently the (beta as of July 2018) Firebase Firestore. These similarly branded offerings have important feature differences, and the latter appears likely to be the recommended choice in the future.

Firestore is a globally scalable, fully managed, document oriented NoSQL database. It is suitable for a very small team to build an application which could then scale to a vast user base with very little system administration work; of course, there are feature trade-offs which enable these amazing properties. Notably, Firestore has important structural limits on the types of queries that can be performed. For example, it has no joins, no “OR” criteria, and limited range (inequality) queries. As I understand, these limitations are what make it possible to engineer Firestore operations to “cost” (and therefore be priced!) in proportion to the amount of data returned.

We recently implemented a Firestore application (with Angular, Angularfire2, and Firebase Functions) in which the query limitations initially were an obstacle; but we found solutions capable of producing great results nonetheless.

Querying workflow state

There are countless scenarios for querying a data store, of course. A simple, common such scenario is an application in which each “document” represents an entity that moves through a workflow over its lifespan. The problem domain doesn’t matter; but one common concrete example is order workflow in a e-commerce system. Such a workflow could look something like this:

New -> Verified -> Scheduled -> In progress -> Preparing -> Shipped -> Delivered -> Closed

Or more generally, think of a workflow feature as the movement of an entity through a series of states:

S1 -> S2 -> S3 -> S4

(In both examples I have drawn simple, linear flows – most of our production/customer software has workflows with looping, branching, and other complex considerations.)

In a system with workflow features, it is very common to need to query a group of entities which are in a certain state – and also very common to query the entities within a set of states. For example, a feature might tally or otherwise view “all orders that have not yet shipped”, which comprises 5 states in the example workflow above.

Firestore “schema” design

Given the absence of OR queries in Firestore, and the frequent need to query entities that are in one state OR another, how should we represent workflow state in a Firestore implementation?

With a traditional, relational database, the main driver of data modeling is to concisely represent the underlying data and ideally “make illegal states unrepresentable”. A representation oriented for this kind of database should follow one of the normal forms, mathematically justified decades ago.

With Firestore (as with most NoSQL data stores), data modeling has a different main driver. An application instead stores data in a way to enable whatever kinds of queries are needed, given the query capabilities. This potentially implies a significantly different data layout, although we often start with something similar to a traditional RDBMS schema then diverge as needed.

Keeping that in mind, show should we store “what state is this entity in?” in a document in Firestore?

Approach 1: Single state field

The simplest solution is one field to represent the state of the entity represented by a Firestore document. The data storage looks something like this:

state: ‘Scheduled’

Querying entities in a single state is trivial with such representation:

.where(‘state’, ‘==’, ‘Scheduled’)

Querying entities in several states requires running a separate query per state, then combining the results together in client code.

Unfortunately, this “combine results in client code” approach, though mentioned in the documentation, has unpleasant consequences. Consider a case where there are 500 entities in state S1, and 500 more entities and state S2, along with some other fields (perhaps “due date”) that ranks all of the entities. Then try to write a query (or pair of queries) to retrieve the 500 “soonest” entities that are in either of these two states:

.where(‘state’, ‘==’, ‘S1’).orderBy(‘dueDate’).limit(500);

.where(‘state’, ‘==’, ‘S2’).orderBy(‘dueDate’).limit(500);

(Your client code would combine the results, sort by due date, and discard all but the first 500 of the combined list.)

Unfortunately, this query now potentially “costs” twice as much, in both time and money, as it should; it may query up to 1000 documents only to discard 500 of them. Still, with this extra cost, time, and client-side query implementation, the single state field approach does work.

Approach 2: One field per state

With this next approach, entity state is represented by a set of flags, one for each state. Typical data could look something like this:

state: {
    New: false
    Verified: false
    Scheduled: true
    InProgress: false
    Preparing: false
    Shipped: false
    Delivered: false
    Closed: false
}

This is more verbose, but easy to understand and implement. Querying for a single state is as easy as before:

.where(‘state.Scheduled’, ‘==’, true)

At first glance, the limitation on OR queries appears to stymie a multiple-state query with this schema. However, with a bit of Boolean logic these OR operations can be swapped out for ANDs and NOTs in the right combination. Transform the desired OR of all the states you want to exclude – into a query which instead excludes all the states you want to exclude.

Continuing with the order-management example, imagine we want all of the orders in the first three states. The query looks something like this:

.where(‘state.InProgress', '==', false)

.where(‘state.Preparing', '==', false)

.where(‘state.Shipped', '==', false)

.where(‘state.Delivered', '==', false)

.where(‘state.Closed', '==', false)

This is simple to implement, mechanically. A bit of utility code could perform the correct set of WHERE operations, leaving application code straightforward.

How might this scale? This is an unknown – these are a lot of ANDs (especially for an entity with many states), which might need more Firestore indexes, or might stress Firestore query mechanism in unexpected ways, or might hit a limit on the number of allowable (or advisable) ANDs.

This is probably the best approach for a small number of states.

Approach 3: Combinatorial state fields

Given how well mathematical logic worked in the previous approach, what if we took it further? When writing an updated state of an entity/document, application code (utility code) could emit all of the combinations of states that include the current state. Concretely, consider the S1/S2/S3/S4 example. If an entity is in state S2, that could be represented like so:

state: {
    S1: false, // or omit the ‘false’ entries
    S2: true,
    S3: false,
    S4: false,
    S1_S2: true,
    S1_S3: false,
    S1_S4: false,
    S2_S3: true,
    S2_S4: true,
    S3_S4: false,
    S1_S2_S3: true,
    S1_S2_S4: true,
    S1_S3_S4: false,
    S2_S3_S4: false,
    // S1_S2_S3_S4 not necessary, would always be true
}

This approach pre-computes the answers to all possible queries of sets of states. Such a representation could be generated easily and consistently by utility code. Queries, again with the bit of utility code to generate them, can find all documents (entities) in any set of states with a single WHERE. For example, to look for all documents that are in state S2 or S3:

.where(‘state.S2_S3’, '==', true)

This will have efficient query characteristics, but could run into limitations around the number of allowable fields in a single Firestore document. Also, with Firestore there is an index for each of these fields, and each index increases the Firestore storage costs and makes document updates a bit slower.

Approach 4: Partial combinatorial state fields

This approach is like the previous approach, with one optimization: rather than pre-generate all of the possible combinations, instead only generate those combinations which enable queries the application actually performs. Typically an application will only use a a few handfuls of combinations of states that an application ever queries for – vastly fewer than the number of combinations of states – which as you may recall from math courses, involves factorials.

The obvious downside: if an application later needs to query a different combination of states, it must first update every document to add the new bit of data, or fall back to another approach.

Approach 5: State range with a numeric field

Up to this point, every approach would work with any kind of workflow, regardless of its linearity. But many applications have a primarily or entirely linear workflow – and that can be used to ease the query challenge. Consider a simple numerical indicator of the state:

state: 3 // entity is in state S3

This enables state “range” queries, like so:

.where(‘state’, '<=', 2) // Entities in state S1 or S2

.where(‘state’, '>=', 3) // Entities in state S3 or S4.

It’s also possible to query any contiguous range of states:

.where(‘state’, '>=', 2)

.where(‘state’, '<=', 3)

Unfortunately, this approach has a serious downside: a Firestore query can only do range queries on a single field. Therefore with this approach, it is impossible to query questions like “entities in state S2 or S3, ordered by due date, first 50”, because the single range query “slot” is used up by the state part of the query. Because of this obstacle, it’s hard to recommend this approach.

Approach 6: State range with boolean fields

Fortunately, there is another variation of the range idea that is quite workable. Use a flag for each state, marked as true if the entity is at or after that state. So for example, an entity at state S3 would be like so:

state: {
    S1: true,
    S2: true,
    S3: true,
    S4: false,
}

Each flag represents a range of states; as before, utility code could implement this representation generically and reliably. Conceptually, a single WHERE can find all entities before or after a point in the workflow. For example, these queries split the state space before/after the S2/S3 boundary:

.where(‘state.S3’, '==', false) // Entities in state S1 or S2

.where(‘state.S3’, '==', true) // Entities in state S3 or S4.

A pair of WHERE clauses (implicitly ANDed) can find all the entities in any contiguous range of states. For example, to find all entities in S2 or S3:

.where(‘state.S2’, '==', true)

.where(‘state.S4’, '==', false)

This approach both avoids an factorial expansion in the number of state variables, avoids major query complexity, and keeps the inequality-query “slot” available for other uses, and is therefore likely an excellent choice if states are mostly linear.

Conclusions and the future

For an application being implemented today on Firestore, which has workflow scenarios where the lack of OR queries is an obstacle, likely one of the above approaches will do a sufficient job. Looking forward to the future, it seems likely the Firestore team will find ways to expand the query capabilities without loss of performance – perhaps rendering these approaches obsolete.

 

Angular Boot Camp Unleashed

Oasis Digital is pleased to announce that…

we are publishing extensive example code that we use in Angular Boot Camp. This example code is available under an open source license (in case you want to grab a bit to use in a project), and is hosted on GitHub for easy browsing and instant editing on StackBlitz:

https://github.com/AngularBootCamp/abc

We’ve published 49 examples so far, with more coming. Why are we publishing this?

  • For students to peruse before class, to better understand what we teach.
  • For students to review after class, as a reminder of what they learned, and to grab code snippets.
  • To provide working, up to date, concise examples of Angular concepts for anyone in the community who needs them.

Here’s a one minute video showing just how easy it is to browse the examples, run them, and view/edit the code:

We have some FAQs also. If you are interested in learning Angular deeply, please consider our class, Angular Boot Camp.

Printable reports in a Node application

Imagine your shiny new web application, JavaScript from end to end (perhaps Node plus Angular/React/Vue/etc) offers a great set of features and a highly interactive user interface. Then a key decision-maker wanders by to praise the interactive features and ask where they click to obtain detailed printable reports like those generated by all the predecessor systems for the last few decades. Uhoh.

It turns out that report still matter, sometimes they land on paper, other times just as PDF files easily passed around without access to the original application. Here are our thoughts on how to effectively generate reports from a Node application. There are many options, but these are what we most commonly see and use.

1: Print the relevant application page

By far the simplest way to get a “report” from a web application is to use the browser’s built-in print capability. To “print a report”, the user navigates to where they see the data they wish to print, then they choose print in the browser. That’s it.

Many pages yield a somewhat poor report from that output by default, but a CSS media-query print stylesheet can rearrange things enough to produce passable results for simple cases. We recommend setting up such print stylesheets, and trying to print pages that have a report like nature, even if another reporting technique is also used – the ability to print a web page has been in browsers since nearly the beginning, and offers a low-cost way to get more value from the same application.

2: Headless-browser-based reporting

Printing an application page in the browser means being limited to whatever HTML is relevant to display in the browser, and further being subject to the vagaries of different browsers. Instead, it’s possible to reuse the same tools (HTML, templating, CSS) on the server to generate specific content for report printing.

To do this, choose any of the highly numerous node HTML templating systems, perform data access however you do so for application features, gather up the data for report, and emit the HTML/CSS. Then use a headless browser (on the server) to transform that HTML and CSS to a PDF, then make it available for the user to download.

This has some compelling advantages:

  • Familiar programming model – as a developer you use approximately the same tools for report output that you use for screen output
  • As a result, it’s easy to get started with relatively little to learn
  • Well suited to reports that generally feel like “documents”, such as dunning letter reports.

Disadvantages:

  • HTML, and therefore this approach, have little in the way of traditional reporting software features
  • Layout tooling for HTML is screen oriented, not report oriented

Implementations include:

3: Reporting library / API

There are Node libraries which offer an API generally suitable to create reports. Unlike the HTML approach, a reporting centric API will have features directly suited for report output, such as creating tabular data output aligned by decimal point, looping over data to put rows in such a table, and so on. This typically will be considerably more concise than HTML approach, because the API will be closer to the problem domain (the domain of “make a report”).

A notable disadvantage here is that the vocabulary of report such an API can produce tends to be much more limited. This is inherent in abstraction, the higher level an API, the easier it is to produce results, but the more constrained the results.

Implementations include:

4: Low-level PDF drawing API

Rather than a high-level API, and there are also low-level APIs available to programmatically create PDF files. A low-level API will have operations like “place text on the page with the following formatting” and “draw a line from coordinate to coordinate”. This low-level, full control means that any report or other output can be produced, but the coding effort to do so can be significant.

This approach typically makes sense only for cases with very specific reporting needs. It is too labor-intensive to create numerous reports this way.

Implementations include:

5: Call a traditional report tool

Lastly, reporting can be thought of as a separate subsystem, whose implementation need not be bound into the same platform as the rest of a system. With this approach, reporting functionality is generally omitted from the application backend, and instead implemented using an off-the-shelf report tool. There is a busy “enterprise” reporting tool market, with multiple very mature products. Costs vary widely, but these tools provide the kind of reporting experience developers may remember from years past: a visual report design surface, a way of interactively running and tweaking and drilling into a report, and so on.

Advantages:

  • Extensive layout possibilities
  • Visual layout tools, rather than code only
  • Report design can often be done by non-developers
  • The reporting solution may offer tools for managing and customizing a large library of reports, out-of-the-box

Disadvantages:

  • Separate subsystem, different tool for staff to learn
  • Deployment complexity, of deploying an additional product rather than only adding a library
  • Cost: some of these are very “enterprise” products, with prices to match

Implementations include:

Choosing an approach

The general trade-off among these approaches is that both cost and capability increase in roughly the order presented above. An acceptable CSS print-specific-stylesheet might yield acceptable results in a few minutes; a separate reporting subsystem, with a new tool stack to learn, could involve a team laboring for months. Oasis Digital has used each approach above (except the headless broader approach, as headless browsers just recently became popular) with excellent results.

CSS grid with Angular and CLI – the time is now

Today, early December 2017, is the time to begin using CSS grid for layout in Angular applications, even if they must support Internet Explorer. We can stop enduring the costs and delays of old “float” based CSS layout, and get better results with less work, using CSS Grid – even with Internet Explorer support requirements – with caveats described below.

Take a look at a running example on the browser of your choice, including both modern browsers and IE11.

https://oasisdigital.github.io/cli-css-grid-demo/

Background

If you’re not familiar with CSS grid, the best source is Rachel Andrew, the global guru of CSS grid. Either read and all of her Grid content, or peruse the links below (thanks mostly to Bill Odom, our early CSS Grid cheerleader, for gathering these). Now is a good time to read and watch, I’ll wait.

Welcome back, CSS Grid fan. Of course the big problem with Grid today is that while support is excellent among current browsers, many users (especially paying, enterprise users) are still wallowing in Internet Explorer. IE has basic support for CSS Grid, but the support is for an older spec which has both fewer features and different syntax. The syntax is irritatingly different enough that manually maintaining both is prone to error.

Fortunately, the incredible Autoprefixer does a very good job, in version 7, of papering over the syntactic differences. In many cases the benefits of Grid can be obtained even without the newer semantics.

Yak shave

Unfortunately, Angular CLI (as of version 1.6.1, as I write this) uses Autoprefixer 6, and exposes no way to adjust Autoprefixer settings. This CLI issue tracker has many open issues, and it appears the team is closely focused on core application bundling and ergonomic considerations, so it’s hard to predict when CLI team attention could turn to issues like this.

Yet here at Oasis Digital, we are ready to use Grid today, and our customers are ready to deploy software today. Therefore, a series of workarounds is in order. To see them in action and in detail, visit this demo repository:

https://github.com/OasisDigital/cli-css-grid-demo

…and for an explanation, read on.

Upgrading Autoprefixer

To use version 7 in an Angular CLI app today, a way must be found to override the Autoprefixer dependency. The traditional answer to override dependencies and settings in CLI is to “eject” – but that is a big leap, not easily reversed, and not recommended. An application based on ejected CLI presents a greater maintenance burden for developers. Instead, we generally recommend sticking with CLI but applying whatever patches are needed at run time to get the right behavior.

Unfortunately as of late 2017, it is still unduly difficult to override dependencies with NPM; searches looking for the way to do it, lead in circles toward old NPM versions. Happily, Yarn can do it quite easily, as a first-class feature. Switch to Yarn, then add a section like this to the package.json file:

"resolutions": {
  "autoprefixer": "^7.2.3"
}

Turn on grid support

Next, Angular CLI does not yet provide a way to pass options to Autoprefixer, and using Grid requires turning the support on. To work past this, the venerable approach of “monkey patch in a postinstall script” solves the problem easily. The script content is essentially just:

sed -i.bak -e 's/autoprefixer()/autoprefixer({grid:true})/' \
 node_modules/@angular/cli/models/webpack-configs/styles.js

This reaches into the relevant file inside the installed CLI code, and edits it in place. I think of this as a rough but necessary hack, to deliver value today, reaching ahead to the future when the tools will make the hack unnecessary.

Fortunately, between these two workarounds there are just a few lines of edit needed in a project. Study the repository above (especially the second commit in the commit history) to see the exact changes.

Conclusion and caveats

Of course there are caveats here, explained in depth by the Rachel Andrew page I linked above. The situation is not quite as severe as that page suggests though, because of what Autoprefixer does. With this setup, you can use today’s Grid syntax, but the subset of Grid semantics supported by IE. This means:

  • Use modern grid-column definitions etc., no need for the older “span” concept.
  • No “flow” in to grid cells – assign grid locations manually. Fortunately, for application layout Flow manual assignment is common anyway.
  • No “gaps” – leave an empty track instead. Easily done.
  • No grid-template-areas.
  • As always, remember to test on IE.

While these caveats are a bit frustrating (especially the lack of grid-template-area), this use of Grid is still an enormous improvement over legacy CSS approaches for many (or most) application screen layouts. With this approach, I see no further reason to wait to start using Grid broadly in Angular applications.

Future work

If the lack of grid-template-areas proves too frustrating, I may look at a similar approach to squeeze in support for postcss-grid-kiss; it provides syntax far beyond that offered by grid-template-areas, and also provides more semantics on IE through use of greater CSS contortions.

 

Angular routing – advice for real applications

There are plenty of examples and documentation about the Angular router, but these things sometimes leave important questions unaddressed.  Documentation often intentionally demures from questions like “what is the best way to use this?”. Even my own previous post briefly reintroducing the router does the same.

Here are our recommendations from extensive use (at Oasis Digital, in classes and complex customer applications), with my specific take on contentious points, on that category of Routing question. How can the built-in capabilities of Angular, including the router, be used with maximum leverage? How can an application be written “with the grain” of Angular to produce the greatest value with the least code? How can the router be used to provide a good user experience and functionality?

URL/route for navigational state

The standard use of intra-application URLs is to represent and control navigational state. Navigational state means “where” the user is in the application. Which screen; which entity; what they are working on; what they are looking at. This type of state so strongly belongs in the URL that (in a polished, important application) it should always be managed via the router -even if some other state mechanism is being used to manage other aspects of application state.

Pop-ups and auxiliary routes

The Angular router has an auxiliary route feature, uncommon among other routers for other frameworks. This feature has various uses, particularly for (unusual) applications with more than one section of the screen that might be navigated separately. But it also has a common use: if an application has a pop-up/popover/dialogue of some kind (for example, a list of users in which editing a user happens on the same screen), the state of whether a pop-up is currently visible should be represented as an auxiliary route.

Resist the temptation to have a pop-up work separately from the route, because that would mean that bookmarking or sharing a URL would not capture this aspect of the user’s navigational state.

Router state and form state

Sometimes a form is used for data entry; for these cases the state of the form (particularly if you’re using model driven/reactive forms) is a fine place to keep that interim data entry state.

But in other cases, a form is used for something like a faceted search. Search parameters can easily stray into navigational state. For example, if the user is currently searching a list of orders for a certain date range that mention a certain product, they could very reasonably want to navigate forward and back to that state, they could want to bookmark and share the URL, and have those search parameters come along.

In these cases, it is reasonable to mutually interconnect the router state and the state of a form. That sounds difficult, but requires just a few lines of code. The result can easily provide a near ideal user experience around searching, URLs, the back button, bookmarks, and so on.

Router state and ngrx/Store

Ngrx/Store users have some extra tools at their disposal around the router state. There is an optional add-on package which integrates router state into Store state, so that it can be managed via the same mechanisms (actions, reducers, effects, etc. An application of significant complexity, so much so that it needs Store, almost certainly also has significant navigational state, and should strongly consider integrating them together.

Don’t fear ugly URL parameters

In simple cases, a URL contains a flat name-value pair list of optional parameters, in which the contents of such strings are most typically just a single value, it is also acceptable to pack in many values in such a single parameter by encoding a broader swath of state as JSON. For example, consider a simple search of orders in the system for order management. It might have single search parameter, perhaps which matches a product description. The URL for the state of searching for such a description could look like:

/orders/search?productMatch=blue

But for a more complex search (for example think of a faceted search with 15 different fields by which the user could search for old orders), you may need more (bug-hiding) code to shuffle search parameters into and out of URL parameters. It is also acceptable, and sometimes more advisable, to encode all of the complex search parameters like so:

/orders/search?q=...

where … Represents a URL-encoded JSON object describing the search parameters

Such a URL is less straightforward to inspect by hand, but also less work to manipulate programmatically and easier to expand to encompass more parameters. Make the trade-off at the application level, as to whether this yields a better overall system.

Router security concerns

I’ve seen suggestions of route guards is a security mechanism; but it’s important to remember that the entire browser is a user agent, it is literally an agent of the user, not the agent of the developer or of the backend system. At best a browser application can avoid making security worse, but it doesn’t actually provide security. Never assume that route guards or other client-side mechanisms are providing any real security, rather think of these mechanisms as advisory security. Advisory security is UX/UI which makes it easier for the user to avoid wandering into a screen which will break because server-side security rules interfere with its operation.

But there is a new and interesting way that browser-based applications can get things wrong with security where the router is concerned. The entire route URL, which means all route segments, parameters, outlets, etc. is untrusted user input. It could accidentally or intentionally contain errant or malicious data. Make sure to treat route data as such, sanitizing it etc. as one would any other user input.

Matrix parameters

Although not used very widely, there is a URL pattern called a matrix parameter, in which each “segment” of the URL has its own parameters rather than just one single bucket of parameters for the entire URL. The Angular router supports this nicely, by using it you can sometimes conserve application code quite significantly while still providing a more ideal user experience around navigational state captured in the URL.

Route guards for data loading

Longtime Angular users who started with AngularJS often point back to the “route resolve” feature is a critical capability they’re looking for an Angular. The Resolve feature makes it possible to delay (or cancel/fail) loading of a route until the data needed to populate the screen for the route is ready.

I recommend using this feature with caution and sparingly. Often a better user experience can be achieved by proceeding directly to a route (for example, a customer history detail display), and then asynchronously loading various parts of the data which appear on that screen. While the screen painting can be a bit messier this way, the user will perceive that the screen started loading much more quickly than if loading is delayed until all data is available. Even if the difference is only a few hundred milliseconds, showing the user partial results is typically a better default.

Angular routing, a basic Q&A

At Angular Boot Camp, we thoroughly introduce and teach the Angular router – over the course of 3 days, spread out into relevant bits and pieces of other learning. Outside of class though, customers ask a straightforward question: What is the Angular Router, and why should I care?

To answer that, this post is a tidy re-introduction to routing in Angular. It is presented in Q&A form – there is little reason to reproduce the router documentation, so this is more like the average of many conversations.

What is routing?

Most concisely, in a web application routing means the relationship between the URL and the state of the application. State can mean a lot of things, but in this context it means “what screen the user is looking at” and “what specific entity/data the user is looking at on that screen”. For example, an application might have “/orders”in the URL when they are looking at a list of orders, or “/orders/12345” when they are looking at order number 12345.

Why use a router?

Routing is about translating between this concise string in a URL, and the rest of the machinery of an application, without coding that translation “by hand”. Developers sometimes ask why they need a thing called a router to do that, whether they might just instead inspect the “window.location” variable and make the application show the right thing. In a sense, the answer is yes – you could certainly do that. But it tends to get complex as an application grows, and if you do it ad hoc, your code won’t have as much in common with other application code. By using a router, you can write less code, and have a standard off-the-shelf solution to a problem that most applications need to solve.

Why care about the URL?

As you learn Angular, you can see how to use variables and ngIfs to make different data appear on the screen in response to user clicks. For example, you could have a variable “orderScreen” and some section of your template using ngIf to display only if orderScreen==true; then have a button which sets orderScreen=true. So you can easily see how to display different data based on what the user clicks, without caring about the URL.

But URLs (and by this I mean the part after the domain name) are the standard, well proven Web way of expressing “where” the user is. Users understand URLs, and users can copy and paste URLs in email, users can bookmark URLs. If your application has a specific URL to mean “order list screen”, a user could bookmark that and navigate directly to it when they like. Fundamentally, URLs are user-friendly.

Would it be easier to just write a separate “application” for my orders list screen? Could I avoid having to understand the router by making each screen a separate application?

I’ve seen applications, especially those adapted to fit inside an server-side system, which eschew the notion of “routing” between different screens and instead have an entirely separate application for each screen. This is possible, but inefficient. The browser ends up needlessly reloading much of the same JavaScript as the user navigates from one screen to another. With the router, the user will only need to load the new, different code for the next screen as they navigate.

Similarly, the router implements “lazy loading”, so just like the idea of totally separate applications, with the router, a user doesn’t have to wait for their browser to load the “order screen” JavaScript until they are ready to use that screen.

The router provides the ideal mix of efficiency in development, efficiency and deployment.

How do I get started with the Angular router?

As you create a new application with Angular CLI, there is a routing option which sets up the basic structure of routing for you. Unfortunately as of late 2017, you still need to manually code up specific routes, which you can do by following the Angular router documentation or various tutorials online (or of course, learn in our class). I expect a future evolution of the CLI will automate more of the router configuration process.

When should I get started with the Angular router?

A few years ago, I used to recommend waiting for routing until you really need it, until your application has more than one “screen”. But now it seems more advisable to simply follow the standard patterns for routing from the very beginning. As you create your first screen in an Angular application, go ahead and implement that screen in a module, and use router lazy loading the load that one module. This seems like extra structure, but will save you from having to rearrange your application code when a second “screen” is inevitably needed down the road. This is also exactly the path we teach in Angular Boot Camp.

What about that idea of routing to a specific (for example) order, rather than to the list of orders?

The idea of routing to a specific individual entity in your application problem domain, use a route parameter. A route parameter is simply a section of a route which can be filled in at runtime with a string. For example, “/orders/12345” suggests a routing set up where the second segment of the route (12345” is a parameter. This is easily configured in your Angular routing configuration, you can see the documentation for the exact syntax.

The more interesting part of a route parameter is consuming that parameters, being aware of it from inside application code. These route parameters arrive at your application component as an observable value. You’ll need to use a small amount of RxJS code to trigger loading of the appropriate data based on that route parameter. This sounds confusing and complex, but you can find examples online would show it is often just a few lines of code.

How do I link to a route?

You can link to a Angular route in an ordinary anchor element (“<A….”) in a component template. You do this using the router link directive (attribute). The documentation shows the exact syntax, but the important thing here is simply that you link to a route within the same application, with the syntax only mildly different from linking to any other page on the Internet.

Things get slightly more complex when you want to link to a specific entity (going back to our example, “/orders/12345”). To do this you use something called a route parameter array, in which the application code snippet has an array with these two parts of the route (orders, and 12345), which then get assembled automatically by angular routing into a working route link.

Of course in real application, users often click a button to do something rather than follow an ordinary web link; you can accommodate this with routing either by styling the link to look like a button (quite easy with bootstrap, for example) or with a line of code in a click handler to ask the router to navigate to a link.

So my links could be navigation in a sidebar or top bar, right?

Yes, the most common use for router links is in a navigation bar of some kind.

In this context, it also makes sense to visually mark which route link is currently “active”. To make it obvious to the user which part of the application they have already navigated to. The Angular router also makes this quite easy with an attribute “routerLinkActive”.

Is there anything else to know about routing?

There is an abundance of important capabilities in the router beyond this quick Q&A introduction. Past this introduction though, it starts to get a little bit more philosophical, and make sense to study after you are already experienced with basic use of the Angular router.  I will follow up with another post on some of these other routing thoughts.

 

 

Software Demonstration and Project Status: Use Video

At Oasis Digital, custom software projects work at various cadences: weekly, biweekly, or sometimes in variable-length cycles. Regardless, at each interval or milestone it’s important to deliver a comprehensive demonstration and status update for our project customer.

Live demonstrations considered harmful

Unfortunately, the most obvious way to deliver demos and status updates does not work very well:

  • Perform a live, high-stakes demo – Murphy’s Law applies. Systems break during a live demo.
  • Freshly, the first time, making it up as you go along.
  • Think about project status only when asked.
  • Seen only by stakeholders who are able to attend the meeting – often a small subset of the people who care about the demonstration and status

It seems silly to even describe these things, but I’ve seen this poor approach as standard across much of the software development world.

Effective software demonstration and project status delivery

We have refined a much better way to deliver software demonstrations and project status updates. The short answer is, “make a video”. The long answer is to make a comprehensive demonstration and project status update video, deliver it to all interested stakeholders, then have a meeting to discuss the demonstration and states. This results in an easier, deeper, and more thoughtful meeting and also serves stakeholders who can’t attend.

Every demo/status video serves a number of purposes and audiences; so it’s important to cover topics of interest to all kinds of stakeholders, not only the stakeholders most able to attend. At the same time, we don’t recommend creating multiple videos for multiple audiences; that is an unsustainable pace of content production, it takes too much time away from the core work of creating quality software.

Make one medium-length video per cycle (week/biweekly/whatever) to address:

  • Demonstration
  • Project progress summary
  • Upcoming work
  • Key open questions
  • Interesting or important technical details

In this way, each video is of value to both “local” stakeholders (the specific customer team managing the project from day to day) and broader stakeholders across a customer organization.

Next, the nitty-gritty of what goes in to a such a video and how to make it. The agenda should go roughly in this order.

Introduction / Title Slide

Files (including video files_ tend to be misclassified, mislabeled, and misplaced. Someone might open up your demo/status video and not know anything about what’s inside. Therefore, always start with a title slide. That slide should include:

  • Name of the project
  • Name and logo of the customer organization the project is for
  • Date (sometimes just month and year, for slower-paced projects)
  • Name and title of the person making this video (speaking)
  • Name, URL, and logo of the company working on the software (for us, “Oasis Digital”)

While the title slide is visible, briefly introduce yourself. You have only a few seconds of viewer attention; the slide and your introduction should last 10 seconds or at most 15, before you cut to the next section.

Still video is a waste of bandwidth, and drives viewers away. Never let the video stay still while you talk for more than a few seconds.

Demonstration

After that brief introduction, jump right into the demonstration. If you learn only one thing about effective demonstrations, here it is:

Get to the payoff fast.

Don’t wander through a long buildup in which only the most dedicated viewer can reach the important part. Show the payoff, the most important bit, within the first few minutes. Then, go back and explain the rest of the story to give a comprehensive demonstration of use cases.

Your demonstration should bring the viewer through one or more use cases relevant to the work underway. Through these use cases, remind the viewer of the overall purpose and functionality of the software project, and point out the new and changes parts, showing progress.

Demonstrations tend to go wrong, or to waste a lot of time, by default. To produce a quality demonstration:

Practice.

Yes, practice. Jot down a terse outline of what you plan to demonstrate, and practice it a couple of times (with the video recorder running) to get familiar with exactly what will happen. If you see anything urgent to fix while making these practice attempts, you might stop and fix it right then. Then once your practice demo goes well, record the real demo.

In a demonstration of a user interface, text and UI elements be readable. We get the best results by sizing the software and recording a “stage” of 1280×720 pixels. A video that size can easily be played back in a non-full-screen window on a typical computer. If your software under demonstration can’t be used at the small window size (i.e. screens that really only work at 1920 resolution), make sure to boost font sizes.

(Some stakeholders, including quite important ones, might only have an opportunity to watch your demonstration video on their cell phone! Think about font and other element sizes accordingly.)

Lastly, create a demonstration you can be proud of. If your demonstration went badly, discard that recording and do it over. If you have been keeping your demonstrations tight, it won’t cost much time if you occasionally have to discard and try again.  If your demonstration is so long that starting over is unthinkable, make shorter demonstrations more often.

Project status and management update

After demonstrating progress on the software, provide an update on the project. We heartily recommend the following order:

  1. Review at what has been done since the last update; positive progress
  2. Preview at what is coming up next; anticipated progress
  3. Discuss upcoming key questions or issues that could delay or prevent progress

Point 1 is especially important and easily overlooked. We have had projects which were objectively going extremely well: delivering a pile of valuable functionality every week for years on end. But looking back, it’s easy to get in a meeting rut – the tone of a project can be ruined by an inadvertent meeting focus on only what is going wrong. Therefore, before discussing what is coming up in what might go wrong, always briefly summarize what has gone well.

The details of how to show status and upcoming work vary by your methodology and toolset. We most often use Jira, and to talk about these things by scrolling, clicking, and talking about an Agile Board in Jira, often supplemented by a Dashboard. You can do the same with other software, or even with a manual project management system.

Obstacles and questions

Having shown visible progress in the demonstration then talked about project status, you now have the viewers’ attention to deal with challenges. Most likely any obstacles or questions are connected to issues in your project management tracking system; so click back through the relevant ones and discuss these things. Make sure to show the relevant part of the software and the relevant bits in the project management software. (Reminder – never more than a few seconds of still video with just a person talking.)

We have found that our recap of obstacles and questions on video, can be very helpful to our customers representatives. They can show the video to other people in their organization who might be able to help with the obstacle. They can listen as well as read – some people enjoyed listening more than reading. They can arrive for a live meeting, already having thought about the questions and ready to answer.

Technical

The last major section of a demo/status video should dig into any interesting or important technical aspects. Here is the chance to show an IDE or source control tool instead of just the running software or Jira board. Most likely the technical bits worth discussing will concern either recently completed features or features coming up shortly, but sometimes a broader topic might warrant attention.

In our experience, digging into the important technical details can also support rapport and credibility with more kinds of stakeholders. Every organization contains a mix of people most responsive to project management, and others most responsive to technical depth.

Closing

As your video ends, flip back to the title slide and thank the viewer for their attention. As hard as you may have worked (more than the length of the resulting video, sometimes much more), your viewer has also dedicated their limited time to watch. Thank them.

Video and audio production tips

Surprisingly, often the most important aspect of video production is audio. You need a quiet room and a decent quality microphone. The former can be hard to achieve in a busy crowded workspace, but it’s worth the effort. Hide in a conference room. Get a coworker to stand guard at the door.

An amply good microphone costs well under $100. We’ve had good results with various types of headsets (but read more about that later), with Blue Snowball microphones, and with a popular Audio Technica model. All of these are quite inexpensive. Any of them are vastly better than trying to use a laptop’s built in microphone.

Next, screen video. You’ll need appropriate screen video recording software, and you will need to master its configuration. We recommend:

  • ScreenFlow, on OSX
  • OBS, on Windows

Video is about more than just the screen though. If you’ve made it this far into this post, you are ready for perhaps the most important advice of all:

Show your face

A demo/status video is not only about information delivery, it is also about personal connection. Humans are hardwired to connect with other humans while looking at their face. Therefore your face should be visible in the video. Both of the software packages above can easily show your face in a corner of the screen. Do so.  (Back to the headset idea – a headset can provide excellent audio pick up, but then you will be wearing a headset in the video. Therefore the headset is not the best solution for this use.)

Video of your face means you need a camera. Most laptops have an amply good camera built-in (but sit your laptop on a stack of books or something handy – so that the laptop camera is not looking up your nose!). Or add an external webcam (< $100) atop an external monitor for better results.

Speaking of cameras, cameras detect light. Rearrange the lighting in your space (or add a $30 lamp) to get some light on the front of your face during your video recording. Your eyes should not be in shadow.

If your recording software supports it (both of the above mentioned packages do), add a “bug”, a term of art for a partially transparent logo in the corner of the screen. For example, if you decide to put your face in the upper right, then the lower left of the screen could contain your company logo at 50% transparency. A video is a branding opportunity in addition to an information communication opportunity.

Finally, reread the advice earlier in the demonstration section, about font and screen recording sizes. Then read them again. 🙂

Feedback wanted

We have worked out the advice here over years of various attempts to communicate demonstration and status information well. But we surely have much more to learn, and appreciate any feedback readers send. Thanks for getting this far, and good luck in your demonstrations and meetings.

Product Development Launch – Default Software and Practices Stack

Context

Here at Oasis Digital, some of our projects are (approximately) “green field” product development launches. The scope of such a project typically includes some CRUD-like features, but also a complex-behavior feature or two. The effort typically lasts a few weeks or at most a few months, after which work is transitioned to customer developers (or occasionally to longer-term ongoing work here).

During a product development launch, we typically demonstrate:

  • Key goals are around user experience, UI development, etc
  • Key use cases of a system
  • Working software, sufficiently deployable for demonstrations
  • Feasibility and suitability of a technology stack, client and server side

Importantly though, during such a launch effort the long-term viability of the underlying customer vision is not yet fixed nor proven. Rather, a product launch refines the vision and proves the potential value.

Executing a product launch

For the reasons above, it is vital that we execute a product development launch expediently. The process is typically something like so.

  • Understand the vision and goals
  • Collaboratively define some key use cases, and key user experiences
  • Defer as much complexity as possible, outside of these key use cases; don’t let the development launch turn in to just a planning effort
  • Choose off-the-shelf tooling to facilitate quick implementation
  • Define key screen flows for the use cases
  • Consider what data appears on each screen (report, integration, etc.), and the flow of data through the system
  • Define an initial “schema of the system”, iteratively through the launch effort
  • Work on an iterative cadence so that we can get through at least several significant iterations during the short project duration

All of that is just context though; what I really want to talk about here is our default software stack for launching a fresh new project. These are just defaults; they often vary by the needs of a specific project, customer, deployment context, etc.

Client / UI

As of 2017, we generally default to a single page web application powered by Angular. While we also work in React and other tools, Angular is where we have the greatest shared experience (from extensive development work, as well as from teaching Angular Boot Camp) and therefore the greatest immediate collaborative productivity.

Angular is also the technology area where we innovate most. We use it for many projects, we train on it, we follow its development closely, we participate in open source. We attend and sponsor conferences. We are connected with the Angular community.

At the same time, customers coming to us for a product launch are typically most interested in seeing a working user interface that demonstrates their vision. Therefore the greatest share of our work in a product development launch is in the user interface.

Server

Because typically our time is focused primarily on user interface/client-side work, it is important to have a set of highly effective tools with which we can execute well-understood server-side APIs very quickly. Therefore, we default to:

  • Java
  • Spring Boot
  • Spring JPA / HIbernate
  • Various other ancillary related libraries and tools
  • A transaction scripting approach for the handful of complex use cases in a bunch effort

These tools are, perhaps to a 2017 eye, somewhat boring. But they are boring because they are well proven, they work. They very rarely yield, within the scope of initial development, any significant obstacles to delivery. That makes them very well suited for a short-duration effort.

Because these tools are so well proven, and because they permit a mostly declarative implementation approach, the resulting small code base warrant little automated testing at the beginning.  We don’t need tests to show that this stack can correctly implement a RESTful API; if it had any trouble doing so, we would replace the stack, not nitpick it with tests.

(While Java is the typical default, we also frequently use Node and related libraries instead; there is a trade-off here between less mature tools, versus the payoff of using more similar technology between client and server code.)

GraphQL

Sometimes things are not quite as boring as they seem though. If the data to be fetched is complex, we typically pick up GraphQL to slash the code quantity and development time for complex data fetching. Data volumes are usually modest during a short-term launch effort, so a straightforward lazy fetching approach via GraphQL resolvers (which go by another name in some implementations) does the job with little effort. This sometimes results in “N+1” database query operations – a problem to be solved later in development, once the scaling and performance attributes are understood.  GraphQL provides a means to do those optimizations, which we defer until they are needed.

Database

At the database layer, we innovate the least. We typically recommend a common and well-proven relational database. Our default is PostgreSQL, although sometimes customer deployment needs may result in MS SQL Server or another RDBMS.

Product development launch efforts are about speed, so we don’t the database schemas by hand. We define data structures in program code then use the tooling (for example, a JPA implementation) generate the schema. Data migration is also generally not an issue in a short-term launch effort; those come into play in a longer lived project which goes in production with data to preserve across versions.

Deployment

Large, long-term software projects will end up with specialized operations experts who shepherd them through critical infrastructure – but this post is about short product launch projects. These projects need to be made visible for review, demonstration and so on, long before the organizational wheels can turn for serious deployment infrastructure.

Therefore, we typically simply deploy the software for demonstration, to a cloud server instance of some kind (AWS, Google Cloud, Digital Ocean, etc), with minor scripting or tooling to automate deploying new versions frequently (sometimes even at every commit) during development. This is not scalable, and not nearly as automatable as more robust solutions, but it is a perfect starting point for something to put in place right away.

Quality

During a short-duration project effort, we write code quickly, but still keep a close eye on quality. Writing good code typically results in faster progress, beyond the timescale of the day or two.

What about the rest of the tools and practices?

Reading back over this description of the choices we make to launch an effort quickly, you might get the impression that we don’t deploy modern techniques, that we haven’t heard of all of the latest (or decades worth) of buzzwords. On the contrary, we have a full array of additional techniques to apply as a project grows in scope, size, and duration.

Micro Services

For certain projects, a micro service architecture produces numerous benefits. But even the esteemed software architect Martin Fowler suggest starting with a monolith: https://martinfowler.com/bliki/MonolithFirst.html

Unit testing

During a product launch, much of the code is boring ordinary use external libraries, which needs very little unit testing. But a project that grows beyond the initial launch will need lots of unit testing around any logic of complexity or interest.

API testing

As APIs become more complex, they warrant thorough test coverage – so a project that grows will get that coverage.

E2E Testing

A product launch effort typically yields a modest number of screens, undergoing rapid change, therefore unsuitable for browser-based end to end automated testing. Therefore, we skipped that during the launch effort.

We don’t forget though – here at Oasis Digital we are very big fans of automated E2E testing, and have seen it pay off on a daily basis, for most any project that lives more than a few months.

NoSQL

NoSQL can solve a great number of problems, and we recommend this type of data store when it is needed. This rarely occurs in the first month or two of the project when the vision and user experience are still being understood.

CQRS / DDD / ES

We have used these techniques extensively, as you can read about elsewhere on our blog. But we mostly set these skills aside during a fast product launch, these are things that pay off its scale but which can make it hard to get to scale as they are allowed to consume too much time early in a project.

Planning and Methodology

During a short launch effort, planning happens primarily via a whiteboard, or spreadsheet, or similar tool. If the values proven in vision works, a project may grow large enough to warrant more complex planning and project management.

Issue tracking

Although we work extensively with issue tracking technology (our sister company builds add-on products and provide services around Atlassian Jira), during a product launch effort our issue tracking approach is intentionally very lightweight. Issues that won’t get attention during the launch effort, are simply listed somewhere tersely. Issues that need attention right away, typically get attention right away, or at are tracked in some lightweight manner. A product launch effort that lasts only a few weeks to a few months might or might not use a “real” issue tracker in that short time, while an effort that grows to a long-term project will use one extensively for tracking, planning, support, etc.

Much more

This is just a short sampling of practices and how they may apply differently at the beginning of a short effort versus late into a large one.

Frameworks and commercial ecosystems

Or, “why we don’t teach Aurelia”

Here at Oasis Digital and its sister company (Expium), we offer training and services concentrated around various languages and frameworks:

  • Angular
  • TypeScript
  • Node
  • The web platform in general
  • JIRA, Confluence, and other Atlassian products (Expium is an Atlassian Solutions Partner)

There are many reason – technical, history, intentional, and accidental – around how we ended up with this set of technologies as our 2017 training and consulting focus.

I was reminded of one key factor today while watching a video from last year of a talk by Rob Eisenberg. Rob is exceptionally sharp, and seems to have a good sense of taste in designing frameworks for developer satisfaction. But I found myself in disagreement over his thoughts around web framework adoption. Rob argues that frameworks like his (Aurelia) are stronger, better choices to build on than frameworks like Angular and React, because first-party training and support services are available for Aurelia from the makers of Aurelia. This initially seems like a compelling pitch, I can see how it would woo some decision-makers. Here is a snippet of one of the slides along these lines, pointing out first party training as an advantage:

But I think ultimately this works out much less well than Rob describes. Why? Because this first party set of training and consulting offerings leave less space for a thriving commercial ecosystem to develop around a framework.

Let’s look at Angular for example. Here at Oasis Digital, we aim to be a leader among many firms around the world, who provide training, consulting, etc. for Angular. Our customers are quite happy with the availability of these services from many different companies; it reduces their risk and means they can shop around for the best fit. Moreover, because Angular has created opportunities for companies like Oasis Digital, it has facilitated a growing commercial ecosystem revolving around the framework. Much the same applies, for example, to React and Vue.js. This is a virtuous cycle. The non-service-offering core team leaves room for others to provide services, which in turn makes it easier and safer for customers to adopt the framework.

(A second example at Oasis Digital’s sister company Expium: Expium focuses entirely on the Atlassian product suite. While Atlassian offers online video training options, Expium’s offerings include things like live human training that don’t compete directly with Atlassian’s offerings. Atlassian enjoys a thriving commercial ecosystem.)

Of course it would be possible for companies like us to offer training and consulting focused on Aurelia. But we don’t want to do that; we like the people responsible for the framework. If we offered services for Aurelia, we would have an inherently competitive relationship with the company behind Aurelia, vying for the same customer opportunities.

This situation applies to various other frameworks and other technical specialties that we could choose to focus on; with so many choices, it inevitably feels wiser to choose those where we can be allied with the core teams rather than in competition with them.

I believe that overall, this is quite important in understanding why some frameworks gain enormous momentum and others do not. Creating this kind opportunity for a commercial ecosystem is an immense competitive advantage to those companies who can offer a framework without needing to build a business directly around it – like Google and Angular.