NgRx is 40x faster than your code – find out why

When I started using @ngrx/store to hold collections of information, I usually put the data into the store as a JavaScript array. It seemed to be the simplest and most appropriate data structure for the information. However, when @ngrx/entity came out, I saw that it used a different pattern – instead of using the array directly, it converts the array to two data structures; an array of ids and an object map keyed by those ids. Why did they do this? And is there a lesson we can learn for our own code?

Continue reading NgRx is 40x faster than your code – find out why

Building with Bazel preview/beta class – Nov/Dec 2019

At Oasis Digital we now offer a “Building with Bazel” class which thoroughly explains the newly-1.0 Bazel build tool from Google.

1-day preview/beta Bazel class

To prepare and validate our new class curriculum, we are running special 1-day preview / beta version of the class. The special 1-day version is offered once online, and once in-person.

We are presenting approximately the first day of material, around 1/3 to 1/2 of what is listed on the class page linked above.

Continue reading Building with Bazel preview/beta class – Nov/Dec 2019

Advanced, Angular-related training and mentoring

About five years ago (it feels like forever) Oasis Digital started training on Angular. Our flagship course Angular Boot Camp has become quite popular, we’ve taught it many hundreds of times to many thousands of students. For the first few years, this offering was a perfect fit for almost every company that contacted us, as software teams were initially adopting Angular. Over the last few years though, Angular has become mature and robust, and Angular has achieved broad adoption across organizations large and small. Aggregate needs of Angular teams inevitably shift toward bigger scale, more difficult and important uses of the technology.

As a result, our training efforts have substantially pivoted toward more advanced topics.

Continue reading Advanced, Angular-related training and mentoring

Transposing Rows and Columns in ag-grid

Real-world Angular applications often need to present tabular/grid data, and most grids make the most sense when presented with each column representing a certain type of data. For example, on a spreadsheet showing a pay schedule for a loan, the first column could be a date, the second column could be the interest accrued, the next could be the size of the payment, etc.

However, we sometimes need to show data in a transposed format, where the rows instead of the columns need to show a consistent data type. This is a rare case, which is why some major grid libraries like ag-grid don’t provide native support for the feature, but it’s still necessary.

Fortunately, ag-grid gives enough power to developers to be able to transpose data for display, and even to have features like renderers and editors apply by row instead of by column.

Continue reading Transposing Rows and Columns in ag-grid

Thinking hard about a project launch

Here at Oasis Digital, we are always agile, and depending on the project needs, sometimes use Agile (in the “capital A” sense) processes extensively. Yet regardless of agility, iterations, steering, and so on, the planning and decisions made at the beginning (and in the early months) of a project often have profound and very difficult to change consequences later.

Therefore, while we would never argue for the straw-man Waterfall, we aim to think very hard about a project at the beginning.

Continue reading Thinking hard about a project launch

Querying without OR in Firestore

Background

Here at Oasis Digital we have successfully used the Firebase Realtime Database, and more recently the (beta as of July 2018) Firebase Firestore. These similarly branded offerings have important feature differences, and the latter appears likely to be the recommended choice in the future.

Firestore is a globally scalable, fully managed, document oriented NoSQL database. It is suitable for a very small team to build an application which could then scale to a vast user base with very little system administration work; of course, there are feature trade-offs which enable these amazing properties. Notably, Firestore has important structural limits on the types of queries that can be performed. For example, it has no joins, no “OR” criteria, and limited range (inequality) queries. As I understand, these limitations are what make it possible to engineer Firestore operations to “cost” (and therefore be priced!) in proportion to the amount of data returned.

We recently implemented a Firestore application (with Angular, Angularfire2, and Firebase Functions) in which the query limitations initially were an obstacle; but we found solutions capable of producing great results nonetheless.

Querying workflow state

There are countless scenarios for querying a data store, of course. A simple, common such scenario is an application in which each “document” represents an entity that moves through a workflow over its lifespan. The problem domain doesn’t matter; but one common concrete example is order workflow in a e-commerce system. Such a workflow could look something like this:

New -> Verified -> Scheduled -> In progress -> Preparing -> Shipped -> Delivered -> Closed

Or more generally, think of a workflow feature as the movement of an entity through a series of states:

S1 -> S2 -> S3 -> S4

(In both examples I have drawn simple, linear flows – most of our production/customer software has workflows with looping, branching, and other complex considerations.)

In a system with workflow features, it is very common to need to query a group of entities which are in a certain state – and also very common to query the entities within a set of states. For example, a feature might tally or otherwise view “all orders that have not yet shipped”, which comprises 5 states in the example workflow above.

Firestore “schema” design

Given the absence of OR queries in Firestore, and the frequent need to query entities that are in one state OR another, how should we represent workflow state in a Firestore implementation?

With a traditional, relational database, the main driver of data modeling is to concisely represent the underlying data and ideally “make illegal states unrepresentable”. A representation oriented for this kind of database should follow one of the normal forms, mathematically justified decades ago.

With Firestore (as with most NoSQL data stores), data modeling has a different main driver. An application instead stores data in a way to enable whatever kinds of queries are needed, given the query capabilities. This potentially implies a significantly different data layout, although we often start with something similar to a traditional RDBMS schema then diverge as needed.

Keeping that in mind, show should we store “what state is this entity in?” in a document in Firestore?

Approach 1: Single state field

The simplest solution is one field to represent the state of the entity represented by a Firestore document. The data storage looks something like this:

state: ‘Scheduled’

Querying entities in a single state is trivial with such representation:

.where(‘state’, ‘==’, ‘Scheduled’)

Querying entities in several states requires running a separate query per state, then combining the results together in client code.

Unfortunately, this “combine results in client code” approach, though mentioned in the documentation, has unpleasant consequences. Consider a case where there are 500 entities in state S1, and 500 more entities and state S2, along with some other fields (perhaps “due date”) that ranks all of the entities. Then try to write a query (or pair of queries) to retrieve the 500 “soonest” entities that are in either of these two states:

.where(‘state’, ‘==’, ‘S1’).orderBy(‘dueDate’).limit(500);

.where(‘state’, ‘==’, ‘S2’).orderBy(‘dueDate’).limit(500);

(Your client code would combine the results, sort by due date, and discard all but the first 500 of the combined list.)

Unfortunately, this query now potentially “costs” twice as much, in both time and money, as it should; it may query up to 1000 documents only to discard 500 of them. Still, with this extra cost, time, and client-side query implementation, the single state field approach does work.

Approach 2: One field per state

With this next approach, entity state is represented by a set of flags, one for each state. Typical data could look something like this:

state: {
    New: false
    Verified: false
    Scheduled: true
    InProgress: false
    Preparing: false
    Shipped: false
    Delivered: false
    Closed: false
}

This is more verbose, but easy to understand and implement. Querying for a single state is as easy as before:

.where(‘state.Scheduled’, ‘==’, true)

At first glance, the limitation on OR queries appears to stymie a multiple-state query with this schema. However, with a bit of Boolean logic these OR operations can be swapped out for ANDs and NOTs in the right combination. Transform the desired OR of all the states you want to exclude – into a query which instead excludes all the states you want to exclude.

Continuing with the order-management example, imagine we want all of the orders in the first three states. The query looks something like this:

.where(‘state.InProgress', '==', false)

.where(‘state.Preparing', '==', false)

.where(‘state.Shipped', '==', false)

.where(‘state.Delivered', '==', false)

.where(‘state.Closed', '==', false)

This is simple to implement, mechanically. A bit of utility code could perform the correct set of WHERE operations, leaving application code straightforward.

How might this scale? This is an unknown – these are a lot of ANDs (especially for an entity with many states), which might need more Firestore indexes, or might stress Firestore query mechanism in unexpected ways, or might hit a limit on the number of allowable (or advisable) ANDs.

This is probably the best approach for a small number of states.

Approach 3: Combinatorial state fields

Given how well mathematical logic worked in the previous approach, what if we took it further? When writing an updated state of an entity/document, application code (utility code) could emit all of the combinations of states that include the current state. Concretely, consider the S1/S2/S3/S4 example. If an entity is in state S2, that could be represented like so:

state: {
    S1: false, // or omit the ‘false’ entries
    S2: true,
    S3: false,
    S4: false,
    S1_S2: true,
    S1_S3: false,
    S1_S4: false,
    S2_S3: true,
    S2_S4: true,
    S3_S4: false,
    S1_S2_S3: true,
    S1_S2_S4: true,
    S1_S3_S4: false,
    S2_S3_S4: false,
    // S1_S2_S3_S4 not necessary, would always be true
}

This approach pre-computes the answers to all possible queries of sets of states. Such a representation could be generated easily and consistently by utility code. Queries, again with the bit of utility code to generate them, can find all documents (entities) in any set of states with a single WHERE. For example, to look for all documents that are in state S2 or S3:

.where(‘state.S2_S3’, '==', true)

This will have efficient query characteristics, but could run into limitations around the number of allowable fields in a single Firestore document. Also, with Firestore there is an index for each of these fields, and each index increases the Firestore storage costs and makes document updates a bit slower.

Approach 4: Partial combinatorial state fields

This approach is like the previous approach, with one optimization: rather than pre-generate all of the possible combinations, instead only generate those combinations which enable queries the application actually performs. Typically an application will only use a a few handfuls of combinations of states that an application ever queries for – vastly fewer than the number of combinations of states – which as you may recall from math courses, involves factorials.

The obvious downside: if an application later needs to query a different combination of states, it must first update every document to add the new bit of data, or fall back to another approach.

Approach 5: State range with a numeric field

Up to this point, every approach would work with any kind of workflow, regardless of its linearity. But many applications have a primarily or entirely linear workflow – and that can be used to ease the query challenge. Consider a simple numerical indicator of the state:

state: 3 // entity is in state S3

This enables state “range” queries, like so:

.where(‘state’, '<=', 2) // Entities in state S1 or S2

.where(‘state’, '>=', 3) // Entities in state S3 or S4.

It’s also possible to query any contiguous range of states:

.where(‘state’, '>=', 2)

.where(‘state’, '<=', 3)

Unfortunately, this approach has a serious downside: a Firestore query can only do range queries on a single field. Therefore with this approach, it is impossible to query questions like “entities in state S2 or S3, ordered by due date, first 50”, because the single range query “slot” is used up by the state part of the query. Because of this obstacle, it’s hard to recommend this approach.

Approach 6: State range with boolean fields

Fortunately, there is another variation of the range idea that is quite workable. Use a flag for each state, marked as true if the entity is at or after that state. So for example, an entity at state S3 would be like so:

state: {
    S1: true,
    S2: true,
    S3: true,
    S4: false,
}

Each flag represents a range of states; as before, utility code could implement this representation generically and reliably. Conceptually, a single WHERE can find all entities before or after a point in the workflow. For example, these queries split the state space before/after the S2/S3 boundary:

.where(‘state.S3’, '==', false) // Entities in state S1 or S2

.where(‘state.S3’, '==', true) // Entities in state S3 or S4.

A pair of WHERE clauses (implicitly ANDed) can find all the entities in any contiguous range of states. For example, to find all entities in S2 or S3:

.where(‘state.S2’, '==', true)

.where(‘state.S4’, '==', false)

This approach both avoids an factorial expansion in the number of state variables, avoids major query complexity, and keeps the inequality-query “slot” available for other uses, and is therefore likely an excellent choice if states are mostly linear.

Conclusions and the future

For an application being implemented today on Firestore, which has workflow scenarios where the lack of OR queries is an obstacle, likely one of the above approaches will do a sufficient job. Looking forward to the future, it seems likely the Firestore team will find ways to expand the query capabilities without loss of performance – perhaps rendering these approaches obsolete.

 

Writing a Generic Type-Safe ng-bootstrap NgbModal Launcher

For an Angular project for one of our clients, I’ve recently started using ng-bootstrap to implement standard modal dialogs in a ModalService. This service has methods to launch confirmation dialogs, input dialogs, message dialogs, etc; you can see a simplified version of this service on StackBlitz.

An addition to the reusable standard dialogs, we also needed to support custom one-off dialogs, and we wanted to use the same general approach, without adding a bunch of duplicate code. Most of the implementation was straight-forward, but adding type safety to the generic launcher was more interesting. Read below to see how interfaces from TypeScript and Angular made it easy.

Continue reading Writing a Generic Type-Safe ng-bootstrap NgbModal Launcher

ng-conf 2018

ng-conf 2018, one of the two “main events” of the Angular calendar, just wrapped up. As usual Oasis Digital was a sponsor, among more and more companies each year.

Perhaps the most interesting news from the conference came in the first few minutes: the main angular.io website (containing the documentation and other overall information about the framework) now receives 1.25 million unique visitors per month. Of course not all of these are Angular developers, on the other hand there are no doubt many Angular developers who visit the documentation quite rarely. Therefore it seems safe to assume there are at least 1.25 million Angular developers out there – the Angular user community is large and growing fast.

On a closer-to-home note, it is always rewarding to have numerous past students visit our booth, and to see numerous past private class customers visible at the conference as major Angular users.

For more on the technical content, please join us for our Angular Lunch user group meeting this week.

 

 

Oasis Digital developer hiring process

Candidates keep asking: what is the process, to be hired at Oasis Digital as a software developer? I believe our process is solid, realistic, and strikes a good balance between speed and depth. Yet merely knowing what the process is, isn’t some great competitive advantage, there’s no reason to keep it secret. Good results flow from the execution, not the checklist.

As of 2018, here is the typical Oasis Digital hiring “workflow”; sometimes it varies for special cases, people we already have an substantial connection with. It’s best to think of this as a kind of “funnel”. At each stage, we are looking for signs that the candidate would perform well and be great addition to our team – and trying to demonstrate to strong candidates that we offer them an opportunity to thrive.

  1. Initial contact or awareness. Perhaps they see a job post somewhere, or someone refers them to us. Ideally they have a chance to watch the short video on our careers page.  Usually we receive a resume (via a tracking system, like everyone else – so that we don’t lose track of any). Of those, some catch our eye. Those move on to…
  2. We have an initial, short conversation about the candidate’s experience, current situation, what they’re looking for in their next job, qualifications, etc. We ideally have this conversations over a video chat, to provide the candidate the maximum “bandwidth” opportunity to make a good impression about these things. A portion of the candidates move on to…
  3. A longer discussion. This discussion again is ideally over a video chat, and often involves more than one of us at Oasis Digital. If the candidate already happens to be in St. Louis, sometimes we meet in our office or over lunch; but video chat is actually “good enough” even for local candidates, and that sometimes can be scheduled more easily or promptly. Of these candidates, a portion move on to…
  4. Our real interview with a software development candidate is to spend time coding together. Ideally this is in-person in our office; although for an out-of-town candidate it’s possible to do this over a screen sharing session. We try to spend at least an hour on this, sometimes several. Working on some code together is by far the most effective way to understand where a candidate is in their development mastery. It is much more effective, and faster, than the sort of “take-home sample programming assignment” that has become popular in recent years. If this goes well and we are favorably impressed, a candidate might move on to…
  5. A deeper, more traditional “HR style” interview, where we talk about the candidate’s experiences, strengths, weaknesses, goals, and so on in depth. Will the person  the person strengthen our team, and reinforce our values? Are our benefits and compensation attractive? Can an agreeable salary be worked out? If all that goes well, the final stage is…
  6. There is a background check process; our customers often require that developers with access to their materials have a clean background check. Assuming nothing negative pops up…
  7. Successful hire – onboarding begins.

From writing all that, wow, it sounds like a lot! In the best case, it can be executed in a few days of elapsed time, although usually there is not such a rush. Our process is intended to be a less onerous time commitment for everyone involved – yet still provide ample opportunity to get to know each other – compared to what some larger companies go through.

 

Angular Boot Camp Unleashed

Oasis Digital is pleased to announce that…

we are publishing extensive example code that we use in Angular Boot Camp. This example code is available under an open source license (in case you want to grab a bit to use in a project), and is hosted on GitHub for easy browsing and instant editing on StackBlitz:

https://github.com/AngularBootCamp/abc

We’ve published 49 examples so far, with more coming. Why are we publishing this?

  • For students to peruse before class, to better understand what we teach.
  • For students to review after class, as a reminder of what they learned, and to grab code snippets.
  • To provide working, up to date, concise examples of Angular concepts for anyone in the community who needs them.

Here’s a one minute video showing just how easy it is to browse the examples, run them, and view/edit the code:

We have some FAQs also. If you are interested in learning Angular deeply, please consider our class, Angular Boot Camp.