Tutorial

Revisiting Dark patterns with the HTML <dialog> tag

 Published: Feb 8, 2021  (last updated: Feb 12, 2021)~ 1200 words~ 6 minutes reading time

Every once in a while I’ll see frontend developers tweet or blog about how great the HTML dialog tag is and each time I’m reminded me of an old Medium post I made a few years ago about its use.

The good news is in 2021 it’s much harder to use it to crash a user’s browser - in Firefox dialog.showModal is behind a flag so it has to be specifically enabled by users. In all browsers preferred way to use this tag in 2021 is to use the open attribute which is a much more sensible way of using the tag. However, it is still possible to do some things mentioned below and kill browser performance (and it’s still possible to steal focus in Chrome).

I thought it would be a good opportunity to revisit it as I feel it still has issues that makes it a potentially dangerous tag to use on the web without thinking about the implementation. One thing that blogpost (and me pointing it out) does is bring out people to say it’s nothing to do with the platform…

So you discovered you can do shit in the browser an mess with the user
Except that nothing of this is related to the dialog tag, nor is the fact that it’s “built-in”. If the argument is “the built-in dialog should be restricted so that people can’t abuse it”, all you’re doing is making sure abusers use a div (or different elem) as dialog instead.

Certainly, some people didn’t agree - and it would be a valid suggestions to say this is achievable in other ways. However, several years later I still stand by my original points:

  • By default, library modals do not completely steal the users mouse focus even when in a loop of generating modals you can click elsewhere on the page (although of course they always can be made to!)
  • It’s still very easy to kill browser performance very easily, very quickly because there is no limit to the number of dialogs that can be created
  • The <dialog> tag is built into the platform, so there’s no way to stop its use by blocking any offending library code - bad actors have all the tools to do this available in the DOM with a single HTML file.
  • A single API is also a single attack surface - any script injection attacks don’t have to worry about library-specific implementations - they can always rely on document.getElementsByTagName('dialog') to access <dialog> elements on the page.

Over the last few years, users have also been taught bad habits to click any popup in operating systems, or in the browser through the use of Cookie popups, newsletter popups and other generally badly implemented ideas.

Revisiting The Hacks

Now on StackBlitz, I’m using the same demo as before with some updates. The example does the following:

  • Adds a listener over the close button which, when the user hovers over it disables the button and moves the modal.
  • Adds a second close button that when pressed closes the modal but also triggers an undesirable path (such as installing malware because we have a trusted user-triggered Event that allows Web APIs)
  • 2 Buttons that send you into a loop of death in infinitely creating modals that can’t be escaped, and always steals focus.
  • Hijacking the users keyboard, so users cannot use Esc, not can they Tab away from the dialog or use Space to click any buttons.

Overriding the DOM for fun

One issue is that you pretty much have full control over the DOM of the dialog. This is good in many ways, but with great power…

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
function createModal() {
  // ...
  const button = document.createElement("button");
  button.innerText = "Close Me";

  function moveModal() {
    button.setAttribute("disabled", true);
    button.innerText = 'LOL';
    lastX = newModal.style.top || 0;
    lastY = newModal.style.left || 0;
    newModal.style.transform = `translate(${(lastX * Math.random() < 0.5
      ? -1
      : 1) +
    Math.random() * 250}px, ${(lastY * Math.random() < 0.5 ? -1 : 1) +
    Math.random() * 250}px)`;
    button.removeAttribute("disabled");
  }

  button.addEventListener('mouseenter', moveModal);
  // ...
}

With our annoying button, we use this to control the transform style of the modal itself and move it beyond the reach of the user. We add a second button that does close the modal using a call to modal.close() but after that we can trigger any JavaScript we want, such as loading new tabs, running bitcoin miners, searching for aliens, etc.

My argument here is that outside of styling some parts of the look and feel, it should not be possible to mess with the dialog after it has been presented.

Blocking the Escape Route

Of course most users might try press the Escape key in hopes that whatever is on the screen will disappear, but we have that particular route covered.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
const keyEvents = [
  "Backspace",
  "Tab",
  "Enter",
  "Escape",
  "Space",
  "ArrowLeft",
  "ArrowUp",
  "ArrowRight",
  "ArrowDown"
];
document.addEventListener("keyup", event => {
  if (keyEvents.includes(event.code)) {
    event.preventDefault();
    createModal();
    return false;
  }
});

In this case we hijack when we detect the code for the Escape key, as well as any other buttons the user might try to use to escape - and we use it to just keep clicking the open button for creating more modals.

My argument here is when a dialog is displayed, don’t allow JavaScript to hijack the keyboard and stop the user from closing the dialog - this is also bad for accessibility.

Killing the Browser

It’s quite easy to create a stack overflow In 2021 it’s less easy to create a Stack Overflow with the dialog, but it is still possible to slow down the users browser and spike CPU by just calling a setInterval on it:

1
2
3
function handleKill1() {
  setInterval(createModal, 1000);  // 1 second here is generous - but we could do it every 1ms
}

Here we can see that we get a lot of spikes in CPU usage and memory garbage collection is constantly triggered:

A Chrome performance graph showing CPU and memory usage

My argument here is to limit the number of dialogs that can be shown on screen at once - modern browsers already do this for alert - allowing the user to set Don't show popus anymore for this website when it’s abused by developers.

Component Solution

One way around this in your own code is to use the <dialog> tag only within your own components - I’ve created another example with a web component that embeds the dialog in to a ShadowRoot and provides the global a state, so you cannot open more than one at a time and third-party scripts cannot access the internal <dialog>.

Hopefully you enjoy the demo and this post. I’m not against the use of dialogs themselves - but I think providing some way to limit the number of modals that can be shown, and limit what can and cannot be changed within it would provide a deeper layer of secure use across the web platform.

Web Serial API with RxJS - Two-Way Reactive Communication between Browser and Serial Hardware

 Published: Jan 30, 2021  (last updated: Feb 12, 2021)~ 1100 words~ 6 minutes reading time

Version 89 of Chrome and Edge browsers have released the Web Serial API unflagged which means as user it’s now available for general use rather than being locked behind experimental flags (if you’re on an earlier version you can enable Experimental Web Platform features in chrome://flags)

The API allows for communication between the browser and supported serial hardware such as Arduino or RaspberryPi over USB Serial connection - the device registers as available to the browser and a port can be opened.

If you don’t have any hardware to connect to, you can easily test it using a Bluetooth Serial connection - provided your computer has a Bluetooth module you can connect your mobile device to it and use the appropriate software.

Connecting Browser to Hardware

To request access to a device, a call needs to be made to the newly available function navigator.serial.requestPort:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
const startButton = document.getElementById("start");

startButton.addEventListener("click", async event => {
  try {
    const port = await navigator.serial.requestPort();
    // We can now access the serial device by opening it
    // e.g. await port.open({baudRate: 9600})
  } catch (e) {
    // The prompt has been dismissed without selecting a device.
  }
});

This function is part of a set that must be called from a specific set of user interactions such as touch or click events - in the demo after a user gesture such as a button click - you cannot just call requestPort from your code without some kind of user interaction as this will cause a security violation. You also must call it from a location that does not have policy set up to disable this (you can see this in the demo above - if you try run it in the editor it won’t work due to the <iframe> not having the correct policy).

You may also need to install the w3c-web-serial types in your project to make sure you have the available types on the navigator object and global types such as SerialPort.

To get a port, call navigator.serial.requestPort inside the handler - it will return a Promise that contains the port object - you can also wrap it in a try/catch to handle when the user cancels device selection.

The port object once created must be called with the open method - the only required property of the options is the baudRate which is the maximum bits-per-second transferred but there are other options based on the requirements of the device.

Once opened the port can return a ReadableStream and WritableStream which allows data to be passed to and from the device.

Our RxJS Operator

To turn this into an RxJS operator we’ll consume the port and set up the functionality to both read and write to the serial bus. You can read the full source code to see how the final Observable was created, but we’ll cover the important sections below.

Reading from the Serial Bus

Once connected, the serial device can start sending data to us - as it’s a ReadableStream the result will be a UInt8Array.

Here we’ll set up an iterable reader for our stream - while the result is not done and the port is still readable, we’ll continue to read the source and emit it to the subscriber of the Observable. If the reader has completed, or the port has been closed we’ll end this iteration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
await port.open({baudRate: 9600});

const process = async (
  result: ReadableStreamReadResult<Uint8Array>
): Promise<ReadableStreamReadResult<Uint8Array>> => {
  subscriber.next(result.value);
  return !result.done || !port.readable
    ? reader.read().then(process)
    : Promise.resolve(result);
};

if (port.readable) {
  reader = port.readable.getReader();
  reader.read().then(process);
}

As the output of our Observable is a Uint8Array. Depending on your needs you can decode this to the format you need, but in most cases it will be text content - here we can use a TextDecoder to get the value:

1
2
3
4
5
6
7
8
const decoder = new TextDecoder("utf-8");

fromWebSerial(port).pipe(
  tap(value => {
    // Value is a UInt8Array, we can append to a element by decoding it
    outputEl.innerHTML = decoder.decode(value)
  })
).subscribe()

Writing to the Serial Bus

The API also allows for writing data to the device, here we can use another Observable that emits a string and provide it to our function as a source, then we can hook it up to the ports WritableStream.

Instead of directly writing, we will create a TextEncoderStream - this allows us to create a new internal writer that we have more control over - it contains both a reader and writer we use this to connect our sources.

The reader from our encoder will be piped to the ports WritableStream, and the writer passed to toWritableStream which connects the Observable to the writer:

1
2
3
4
5
6
7
8
if (writerSource && port.writable) {
  const encoder = new TextEncoderStream();
  writerEnd = encoder.readable.pipeTo(port.writable);
  const outputStream = encoder.writable;

  writer = outputStream.getWriter();
  writerSource.pipe(toWritableStream(writer, signal)).subscribe();
}

Now we can pass the Observable and use it to emit our values:

1
2
3
4
5
const emitter$ = new Subject<string>();

fromWebSerial(port, emitter$.asObservable()).subscribe();

emitter$.next('Hello There!');

Creating a Serial Chat App

Now we can read from, and write to, our hardware device the possibilities are endless with what we can do - provided the hardware supports it.

For this tutorial I build a very basic chat app - using the Bluetooth Serial applications mentioned above you can use it to send and receive text data between devices.

A screenshot of a web serial mobile app A screenshot of a web serial browser app

In the example code I’ve set up a button to enable our port request - you should see a popup with a list of devices available for you to use. After connecting a basic chat interface will show up - type in some text and check out your device software - you should see the same message there, and you can then send a message back to the browser.

Hopefully you’ve found this tutorial useful and if you do build something with this I’d love to hear about it!

A collection of pre-built operators and Observables for your projects

The RxJS Logo, a Ninja jumping over a moon

RxJS Ninja - is a collection of over 130 operators for working with various types of data (such as arrays , numbers ) and streams allowing for modifying, filtering and querying the data.

Still in active development, you might find useful operators that provide clearer intent for your RxJS code.

You can check out the source code on GitHub .

Create your own Dark Mode Detection Observable using RxJS and Media Queries

 Published: Jan 27, 2021  (last updated: Feb 12, 2021)~ 1100 words~ 5 minutes reading time

Demo Link

One of the more recent features available in browsers is ability to do CSS Media Queries based on user theme & accessibility settings in the operating system - for example using @media (prefers-color-scheme: dark) (see prefers-color-scheme ) you can check if the users OS theme is currently in Dark Mode and use this to set a websites theme accordingly.

The query is also available in JavaScript using the window.matchMedia function - that returns a MediaListQuery that will allow us to do two things:

  • The current value of the users setting via the matches boolean property
  • Any future values by listening to its changes event and attaching an event lister function to it

Combining these, it’s the perfect candidate to turn into a fully reactive dark mode switcher using RxJS and Observables that will give us the users current setting. If you’re not familiar with Observables, they are a type of stream that emits values over time - consumers can subscribe to these Observables to get their values - this means we can use them to get values from long running functions or event emitters.

In the full demo you’ll find an example page with light and dark mode set from your own OS settings. To see the full working example you need to change the setting (e.g. in OSX Dark Mode is under “General” settings) - also provided are a user toggle button, and a button to turn off and on the media query listener. The Observable in this example supports more than one prefers- type of query but in the tutorial below we’ll build a much simpler isDarkMode Observable than the one provided in the demo, but the concept is the same.

Creating a Dark Mode Observable

For our code we first need to create our Observable factory - this is our function that allows us to pass any required parameters for the implementation and returns an Observable which can then be subscribed to.

The Observable constructor takes a function - a callback any time there is a new subscription - this is where the implementation will live.

As soon as the subscription opens, we first check to see if window.matchMedia is available - it should be available in all modern browsers but is not available in environments like node (yay unit testing!) - so here we can throw an error.

The factory also accepts an optional AbortSignal , an object that contains an onabort callback - the parent of the signal is a AbortController and using this we can externally signal our Observable to close all subscriptions and remove all event listeners.

The return value of the constructor is another function - the teardown logic - this is called when an RxJS subscription is ended, such as using takeUntil or take(1) - here we also ensure that all subscriptions and event listeners are closed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
import { Observable } from 'rxjs';

export function isDarkMode(signal?: AbortSignal): Observable<boolean> {
  return new Observable<boolean>(subscriber => {

    if (!window.matchMedia) {
      subscriber.error(new Error('No windows Media Match available'));
    }

    if (signal) {
      signal.onabort = () => {
        !subscriber.closed && subscriber.complete()
      }
    }

    return () => {
      !subscriber.closed && subscriber.complete()
    }
  });
}

Adding the Media Query

The main implementation of our Observable is to create our MediaListQuery and use it to emit values to any subscribers. On creation, contain a matches value of true or false which can be immediately be passed to subscriber.next.

We also need to bind a listener using to the change event of the query. As we also need to remove this later create an internal private function for the event handler - this will also call subscriber.next each time there is a detected change.

Also casting the event to a MediaQueryListEvent ensures TypeScript recognises it has the matches property which contains our value.

1
2
3
4
5
6
7
function emitValue(event: Event) {
  subscriber.next((event as MediaQueryListEvent).matches);
}

const mediaListQuery = window.matchMedia('(prefers-color-scheme: dark)');
mediaListQuery.addEventListener('change', emitValue);
subscriber.next(mediaListQuery.matches);

Cleaning up handlers and subscriptions

Already we can start to use the new Observable, but we also need to make sure that we:

  • End any subscriptions to the Observable when either the AbortSignal fires or RxJS unsubscribes from it
  • Remove any event listeners in the DOM for the change event

With a slight bit of refactoring we have our final Observable factory below - in both the signal.onabort and the Observable teardown logic we remove the event listener - the API for this requires you pass the function implementation from our private function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
import { Observable } from 'rxjs';

export function isDarkMode(signal?: AbortSignal): Observable<boolean> {
  return new Observable<boolean>(subscriber => {

    if (!window.matchMedia) {
      subscriber.error(new Error('No windows Media Match available'));
    }

    function emitValue(event: Event) {
      subscriber.next((event as MediaQueryListEvent).matches);
    }

    const mediaListQuery = window.matchMedia('(prefers-color-scheme: dark)');

    if (signal) {
      signal.onabort = () => {
        mediaListQuery.removeEventListener('change', emitValue)
        !subscriber.closed && subscriber.complete()
      }
    }

    mediaListQuery.addEventListener('change', emitValue);
    subscriber.next(mediaListQuery.matches);

    return () => {
      mediaListQuery.removeEventListener('change', emitValue);
      !subscriber.closed && subscriber.complete()
    }
  })
}

Finishing up

Now we have a fully working reactive Observable for a Dark Mode media query, this can be used in any application or website to check the users theme setting. The demo provides some more example of how to do this in full.

1
2
3
4
5
6
isDarkMode().pipe(
  tap(value => {
    body.classList.removeClass(value ? 'light' : 'dark');
    body.classList.addClass(value ? 'dark' : 'light');
  })
).subscribe()

This tutorial is just one small example of the kind of things that can be done with RxJS - any API that can emit values over time can be turned into Observables.

A collection of pre-built operators and Observables for your projects

The RxJS Logo, a Ninja jumping over a moon

RxJS Ninja - is a collection of over 130 operators for working with various types of data (such as arrays , numbers ) and streams allowing for modifying, filtering and querying the data.

Still in active development, you might find useful operators that provide clearer intent for your RxJS code.

You can check out the source code on GitHub .

Creating Custom RxJS Operators

 Published: Jan 25, 2021  (last updated: Feb 12, 2021)~ 1500 words~ 7 minutes reading time

RxJS is a popular library available for TypeScript and JavaScript.

It provides APIs for the creation of applications and libraries using asynchronous streams of data and reactive methods. It’s one of the foundation libraries of Angular .

Included in it are over 100 operators - functions that take an Observable stream of data and return values for use in chains of operators.

Many of the operators are low level, and combining them through the pipe method they create a powerful way to work with data.

Creating custom operators for a domain

The good news is it’s also very easy to create new higher-level operators for our domain code - these can be used where you find duplicate or complicated operations.

Creating operators we can also ensure well-tested code using marble testing and they can be shared among your team to make your code more readable and stable.

There are two types of operators that can be created - a MonoTypeOperatorFunction and OperatorFunction and all operators must do two things:

  • Return a function which accepts as its parameter a source from the previous Observable value in the stream
  • Return a value of the same type for MonoTypeOperatorFunction or different type for an OperatorFunction by using the source value with pipe

Below we’ll have an example of each, but first, to support creating the operators we need some code to simplify:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
import { from } from 'rxjs';
import { map, tap } from 'rxjs/operators';

// Create a cold source that will emit each number
const source$ = from([1, 2, 3, 4, 5]);

// Create a cold source that multiplies each number by `5`
const multiplyByFive$ = source$.pipe(map(value => value * 5));
// Create a cold source that multiplies each number by `10`
const multiplyByTen$ = source$.pipe(map(value => value * 10));

// Subscribe to the sources and console.log the output
multiplyByFive$.pipe(tap(console.log)).subscribe();
// Output: `5, 10, 15, 20, 25`

multiplyByTen$.pipe(tap(console.log)).subscribe();
// Output: `10, 20, 30, 40, 50`

Creating MonoTypeOperatorFunction for single types

As the name suggests a MonoTypeOperatorFunction is a function that works with a single type of data - the input and output value must be of the same type.

Looking at our code we can identify two multiplication operations in our code that are the same. To turn this into an operator the function will look like this:

1
2
3
4
5
6
import { MonoTypeOperatorFunction } from 'rxjs';
import { map } from 'rxjs/operators';

export function multiply(factor: number): MonoTypeOperatorFunction<number> {
  return (source) => source.pipe(map(value => value * factor))
}

Here, we are returning an arrow function that takes the previous source - which must be an Observable<number>. The source is piped to map which allows the source value to be converted to a new value, in our case we multiply by the factor

TypeScript understands that the output must also be a number - and if you try to return another value type it will throw a compile error.

Writing a marble test

Marble testing is a way to write tests for RxJS operators that deal with data over time - data is not static due to its asynchronous nature and cannot always be guaranteed in a specific order. Luckily the test for this operator is simple.

My personal preference has been to write these test in Jest using rxjs-marbles and jest-marbles , but there are other libraries to write these test available.

Using marbles, we can set up a mock source that will emit 5 numbers at the specified frames.

The test result contains two things:

  • A subscriptions string which is used to check that the operator handle subscription ending properly using toHaveSubscriptions
  • An output Observable that will contain the results of the operator and compared against the expectations using toBeObservable

In this test, we’ll pass a source of numbers and multiply by 10

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import { marbles } from "rxjs-marbles/jest";
import { map } from "rxjs/operators";
import { multiply } from './multiply'

describe("multiply", () => {
  it("should multiply by 10", marbles(m => {
    const input = m.hot('-a-b-c-d-e-|', {a: 2, b: 3, c: 4, d: 5, e: 6});
    const subs = '^----------!';
    const expected = m.cold('-a-b-c-d-e-|', {a: 20, b: 30, c: 40, d: 50, e: 60});
    m.expect(input.pipe(mul(10))).toBeObservable(expected);
    m.expect(input).toHaveSubscriptions(subs);
  }));
});

Update Code

Now the operator is created it can be used in the existing code from above - ideally the operator should be part of a shared library of code:

1
2
3
4
5
6
7
import { from } from 'rxjs';
import { multiply } from '@myorg/rxjs-library'

const source$ = from([1, 2, 3, 4, 5]);

const multiplyByFive$ = source$.pipe(multiply(5));
const multiplyByTen$ = source$.pipe(multiply(10));

Already much more readable! Our code explains our intent, but we haven’t really reduced the duplication of our sources.

Changing the API with OperatorFunction

In our domain, we know we always want more than one value from a source and using the OperatorFunction we can use that to reduce our duplicate code even more.

This would introduce an API change, but with proper tests, we should be able to migrate our code easily.

For our source value, it is still a single number value, but in the API we’ve changed:

  • The input factor can be a single value or an array of values
  • The return value is now an array of values, regardless of the input.

Instead of forcing the users to check the type of response, this single API can be well documented and expected when we use it in our code:

1
2
3
4
5
6
import { OperatorFunction } from 'rxjs';
import { map } from 'rxjs/operators';

export function multiply(factor: number | number[]): OperatorFunction<number, number[]> {
  return source => source.pipe(map(value => (Array.isArray(factor) ? factor : [factor]).map(f => value * f)))
}

Updating the tests

First, we need to update the existing test - here we only have to change the values in our expected Observable - we now expect an array of numbers regardless of the input - but with a single value our array length will be 1

1
2
3
4
5
6
7
it("should multiply by 10", marbles(m => {
  const input = m.hot('-a-b-c-d-e-|', {a: 2, b: 3, c: 4, d: 5, e: 6});
  const subs = '^----------!';
  const expected = m.cold('-a-b-c-d-e-|', {a: [20], b: [30], c: [40], d: [50], e: [60]});
  m.expect(input.pipe(mul(10))).toBeObservable(expected);
  m.expect(input).toHaveSubscriptions(subs);
}));

To ensure full coverage, we should also test for the case where were have an array input for the multiplication factor:

1
2
3
4
5
6
7
it("should multiply by by 5 and 10", marbles(m => {
  const input = m.hot('-a-b-c-d-e-|', {a: 2, b: 3, c: 4, d: 5, e: 6});
  const subs = '^----------!';
  const expected = m.cold('-a-b-c-d-e-|', {a: [10, 20], b: [15, 30], c: [20, 40], d: [25, 50], e: [30, 60]});
  m.expect(input.pipe(mul([5, 10]))).toBeObservable(expected);
  m.expect(input).toHaveSubscriptions(subs);
}));

Update Code

We can now update the code further - here we can now remove the two additional cold Observables and create a single one using our new multiply operator, passing it an array containing out factors:

1
2
3
4
5
6
import { from } from 'rxjs';
import { multiply } from '@myorg/rxjs-library'

const source$ = from([1, 2, 3, 4, 5]);

const multiplyValues$ = source$.pipe(multiply([5, 10]));

Now we can subscribe to the multiplyValues$ source and get both our new result which contains the multiplication of both numbers

1
2
multiplyValues$.pipe(tap(console.log)).subscribe();
// Output: `[5, 10], [10, 20], [15, 30], [20, 40], [25, 50]`

Next Steps

You can see a working version of this operator on StackBlitz by opening the console to see the result.

This operator is just a taste of what’s possible with RxJS - diving into the API you’ll find many more operators to help work with data in other synchronous and asynchronous operations.

A collection of pre-built operators for your projects

The RxJS Logo, a Ninja jumping over a moon

Now for a shameless plug - my own library - RxJS Ninja - is a collection of over 130 operators for working with various types of data (such as arrays or numbers ) and streams allowing for modifying, filtering and querying the data.

Still in active development, you might find useful operators that provide clearer intent for your RxJS code.

You can check out the source code on GitHub . There you can also find a starter project for creating your own TypeScript libraries like this.

Validating data with JSON Schema, Angular and TypeScript

 Published: Sep 18, 2019  (last updated: Feb 12, 2021)~ 1900 words~ 9 minutes reading time

One common question I see with a lot of new TypeScript developers is how to handle runtime validation of data, using the types they have built.

The issue is web platform, as yet, does not support types. Typescript itself is a higher-level language built on top of JavaScript and uses a compiler to create compatible code for the web, node or other JS platforms - this means that types are only available at design time.

Most developers have a method or form in their code where they want to validate data being passed in is correct before sending it to another API. This works for hard coded data in Typescript, but not dynamic data from sources such as a form or API source

The good news is the problem itself has been solved and there are several solutions for TypeScript such as io-ts or joi but I find these solutions to encourage duplication of types across different domains to maintain both your types and validation objects.

Introducing JSON Schema

A much simpler way to maintain both types and validation within a project is to use a single source of truth. The main option for this is JSON Schema .

A JSON Schema file allows you to define types using a JSON file, using a specification defined by the selected draft (at the time of writing it’s number 7 ).

This file can be used to generate types for design-time coding using CLI tools and can be used for data validation at runtime using another library that can consume a schema to generate a validation method.

Schema Example

For this demo, I’ve created a simple schema object defining a customer within a system. The customers’ properties are:

  • ID
  • firstName
  • lastName
  • dateOfBirth
  • email

In this example we set "additionalProperties": false to keep the example simple, but it’s a very flexible option!

If set to true or not included, the outputted types will include an indexable type with a [key: string]: any at the end of the type properties.

You can also pass it properties such as "additionalProperties": { "type": "string" } which will allow only string additional properties to be added.

By setting to false - only the properties defined will be available on the type, which I’ll do for this example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
{
  "$id": "https://tane.dev/customer.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Customer Record",
  "type": "object",
  "properties": {
    "id": {
      "type": "string",
      "description": "The Customers ID in our system"
    },
    "firstName": {
      "type": "string",
      "description": "The customer's first name."
    },
    "lastName": {
      "type": "string",
      "description": "The customer's last name."
    },
    "email": {
      "type": "string",
      "format": "email",
      "description": "The customers email address"
    },
    "dateOfBirth": {
      "type": "string",
      "format": "date",
      "description": "The customer's date of birth."
    }
  },
  "additionalProperties": false,
  "required": [
    "id",
    "firstName",
    "lastName",
    "dateOfBirth",
    "email"
  ]
}

The first tool we will run this through is the imaginatively titled <code>json-schema-to-typescript</code> ! This project will take a valid schema file and generate a file containing the types. From the example above the output is:

json2ts -i customer.json -o customer.d.ts --style.singleQuote

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
/* tslint:disable */
/**
 * This file was automatically generated by json-schema-to-typescript.
 * DO NOT MODIFY IT BY HAND. Instead, modify the source JSONSchema file,
 * and run json-schema-to-typescript to regenerate this file.
 */

export interface CustomerRecord {
  /**
   * The Customers ID in our system
   */
  id: string;
  /**
   * The customer's first name.
   */
  firstName: string;
  /**
   * The customer's last name.
   */
  lastName: string;
  /**
   * The customers email address
   */
  email: string;
  /**
   * The customer's date of birth.
   */
  dateOfBirth: string;
}

One thing to note is that email and dateOfBirth are string in our type, the format is only used with validation. It is possible to create types for these fields and reference them using a more complex schema .

This type can now be imported into other types, and the json-schema-to-typescript will do this when you use complex references. For example, if we define an entire customer order type it might look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
import { CustomerRecord } from './customer';
import { OrderItem, Checkout, Address } from './shop-front'

export interface CustomerOrder {
  customer: CustomerRecord;
  deliveryAddress: Address;
  billingAddress: Address;
  items: OrderItem[]
  checkout: Checkout
}

Also, all the properties have been added to the required array. When creating a new customer, if the data does not contain an ID, you can use the Partial type to accept an incomplete object - if you expect your API to give back a full object you can return a CustomerRecord. You can also use Required where you need to ensure all fields are passed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import { CustomerRecord } from './customer';

class CustomerClass {
  // Return a API request with a customers object
  async addCustomer(customer: Partial<CustomerRecord>): Promise<CustomerRecord> {
    return this.api.save(customer);
  }
  
  // Return a API request with a customers object
  async updateCustomer(customer: Required<CustomerRecord>): Promise<CustomerRecord> {
    return this.api.update(customer);
  }
}

Validating with the Schema

Now you have types, it makes the development of your application easier - but we still need to validate data entered is correct.

One way is to use the same schema on the server-side, using your languages JSON Schema validator, but in this example, I’ll use ajv - a javascript library that allows a schema to be loaded and data validated against it. The documentation is quite complete on using it in a JavaScript environment so I won’t repeat it too much here, but instead, I’ll build an Angular module that can be provided as a schema validation service.

First, we’ll create the Angular module, in to which we inject the AJV class and allow the user to provide a configuration, the service is provided below. This allows the module to be imported with a configuration and a service that is injectable through your application.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import { NgModule, InjectionToken } from '@angular/core';
import { HttpClientModule } from '@angular/common/http'
import { JSONSchemaService, AJV_INSTANCE } from './json-schema.service';
import ajv, { Ajv, Options } from 'ajv';

export const AJV_CLASS = new InjectionToken<Ajv>('The AJV Class Instance');
export const AJV_CONFIG = new InjectionToken<Ajv>('The AJV Class config');

export function createAjvInstance(AjvClass: any, config: Options) {
  return new AjvClass(config);
}

@NgModule({
  import: [HttpClientModule],
  provides: [
    JSONSchemaService,
    { provide: AJV_CLASS, useValue: ajv },
    { provide: AJV_CONFIG, useValue: {} },
    {
      provide: AJV_INSTANCE,
      useFactory: createAjvInstance,
      deps: [AJV_CLASS, AJV_CONFIG]
   }
  ]
})
export class JSONSchemaModule {}

Now we create a service - within this service it will access to the Ajv class that allows the service to be provided with schemas via an Angular HTTP call. The parsed schema is assigned a name and can be used through the app using dependency injection - this service is a good use case of a root service too, which create a Singleton of the service shared within the same application.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
import { Injectable, Inject, InjectionToken } from '@angular/core';
import { HttpClient } from '@angular/common/http';
import { Ajv } from 'ajv';

export const AJV_INSTANCE = new InjectionToken<Ajv>('The AJV Class Instance');

/**
 * The response of a validation result
 */
export interface ValidateResult {
  /**
   * If the result is valid or not
   */
  isValid: boolean;

  /**
   * Error text from the validator
   */
  errorsText: string;
}


@Injectable({
    provideIn: 'root'
})
export class JSONSchemaService {
  constructor(private readonly http: HttpClient, @Inject(AJV_INSTANCE) private readonly ajv: Ajv) {}

  /**
   * Fetches the Schema and adds it to the validator schema set
   * @param name The name of the schema, this will be used as the key to store it
   * @param urlPath The URL path of the schema to load
   */
  public loadSchema(name: string, urlPath: string): void {
    this.http.get(urlPath).subscribe(result => this.ajv.addSchema(result, name));
  }

  /**
   * Validate data against a schema
   * @param name The name of the schema to validate
   * @param data The data to validate
   */
  public validateData<T>(name: string, data: T): ValidateResult {
    const isValid = this.ajv.validate(name, data) as boolean;
    return { isValid, errorsText: this.ajv.errorsText() };
  }
}

Now we can use our service to load a JSON schemas into an internal Ajv map, and using the key load the schema to validate a data object against it. The service could be used alongside a form, any methods on a service or checking the result of one API before passing to another API.

A simple example of how it could be used in a form component (the example is shortened, most likely you would load your schemas from another service) or how you could validate the parameters passed to a method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
@Component({
  selector: 'my-form-component',
  template: `
    <errors-component *ngIf="let errors; errors$ | async"></errors-component>
    <form [formGroup]="customerForm" (ngSubmit)="submit()">
      <!-- Customer form in here --->
    </form>
  ` 
})
export class FormComponent {
  
  error$ = new BehaviorSubject<string>('');

  customerForm = this.fb.group({
    id: [''],
    firstName: [''],
    lastName: [''],
    email: [''],
    dateOfBirth: ['']
  });

  constructor(private readonly fb: FormBuilder, private readonly schema: JSONSchemaService, private readonly app: AppService) {
    this.schema.loadSchema('customer', 'https://tane.dev/customer.json')
  }

  /**
   * In the submit method, we validate the input of a form - this can be on top of, or instead
   * of Angular form validation
   */
  submit() {
    const result = this.schema.validateData('customer', this.customerForm.value);
    if (result.isValid) {
       this.app.updateCustomer(this.customerForm.value);
    } else {
      this.error$.next(result.errorsText);
    }
  }
 
  /**
   * This custom method can take a type of T (which in this case is an `any`) and validate
   * that the data is valid
   */
  customMethod<T = any>(data: T) {
    const result = this.schema.validateData('customer', data);
    if (result.isValid) {
       // Do custom logic
    } else {
      this.error$.next(result.errorsText);
    }
  }
}

Conclusion

I hope you’ve found this article useful in helping understand how and where Typescript can be used to validate data within an application, and JSON Schema to validate dynamic data.

Please feel free to leave feedback on any issues or improvements, but hopefully, these examples give a clearer understanding.

For full documentation of JSON Schema, check out the Understanding JSON Schema pages to get examples of using allOf, anyOf, oneOf and using definitions

Providing injectable features to Angular modules

 Published: Mar 14, 2019  (last updated: Mar 16, 2019)~ 1700 words~ 8 minutes reading time

Working with Angular and Typescript ; as I have refactored and re-written components I’ve been learning to take advantage of one of the more powerful features - dependency injection.

Building Angular components, most developers will have already used this to inject features like the HTTP client or the FormBuilder to be used in a component. A common service example might look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
import { Injectable } from '@angular/core';
import { HttpClient, HttpResponse } from '@angular/common/http';
import { Observable } from 'rxjs';

@Injectable()
export class RequestService {
  constructor(private readonly http: HttpClient) {}

  public getExample(): Observable<HttpResponse> {
    return this.http.get('/api/example');
  }
}

Angular’s own features can be injected via their class identity (e.g. HttpClient). You can also inject any components, services or pipes that are registered within a module.

Angular also provides a mechanism for custom injections, this means items like configurations or instances of external libraries can be passed into any constructor within your module or any item within the dependency tree. To do this the library needs to provide an InjectionToken which represents the key in the dependency tree, and use the Inject decorator to provide an instance into a class constructor (this is covered later in this post).

Creating The Service

I recently worked on refactoring some code where in a component it had some code to handle getting a session object from sessionStorage. When working with Angular’s ng serve making changes will often result in a refresh, destroying any existing session. This feature allowed session storage to be turned on and keep it between refreshes by allowing the component to check if a value existed in sessionStorage object.

Using the principles in SOLID I moved the code into a service. Having a service do one thing it made the code easier to understand. The unit test for this is also smaller and can cover most of the component.

The service has one additional property which is storagePrefix, used to make each instance of the service unique to it’s component.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
import { Injectable } from '@angular/core';

@Injectable()
export class ComponentService {
  private storagePrefix = 'my-app';

  public init(storagePrefix: string) {
    this.storagePrefix = storagePrefix;
  }

  public load(key: string): string | undefined {
    return sessionStorage.getItem(`${this.storagePrefix}-${key}`);
  }

  public save(key: string, value: string): void {
    sessionStorage.setItem(`${this.storagePrefix}-${key}`, value);
  }
}

In it’s current implementation it’s usable, but it has a few issues. The main one is that it uses sessionStorage but trusts that it exists on the global object, this means we should always expect it to be available, and is a bad practice. It also limits this service to only use sessionStorage and no other key/value store. It would be better if we could provide it to our service instead.

We can also set the storagePrefix via our app configuration and means our component doesn’t need to understand this configuration (yet!) so it avoids us having to call the init method within a component’s ngOnInit.

By making these injectable, we will not only be able to provide our sessionStorage but we can support any module or library that conforms to the Web Storage API . This means we no longer need to rely on the global object and we have more control over what we inject.

The powerful feature of dependency injection is the ability to pass instances of other non-angular applications. For example you could pass the instance of a jQuery application and call it’s methods from your application (you’ll also want to check out NgZone when running these, but I’ll discuss that in a future post).

This allows for the transition of applications over time within an organisation, rather than having to replace all applications in one go. It’s a good way to safely inject features from third-party libraries and means passing mock versions is easier for tests.

Creating an injectable service and module

Making features injectable

First we need to define the interface of the object want to use for the configuration. This interface provides a contract in the forRoot method described below and tells developers what options they can provide to the application

Our configuration will require a storagePrefix as a string - this is need to set the name of the store. We also pass our Storage instance, but with this we can set a default value in our component

1
2
3
4
export interface ComponentServiceConfig {
  storagePrefix: string;
  storageInterface?: Storage;
}

Next we will create an InjectionToken, a token that is usually constant representation of something we want to provide via dependency injection. You can use any type of object as your token type, and can even provide a factory method to provide dependencies. Check out the Tree-shakeable Injection Token to see how you can do some advanced use. In our case we will just have the component config object.

1
2
import { InjectionToken } from '@angular/core';
const COMPONENT_SERVICE_CONFIG = new InjectionToken<ComponentServiceConfig>('COMPONENT_SERVICE_CONFIG');

Now we have the token service we will provide a simple configuration with this token which will be an object, but a token can be used to inject anything from a string to a function, or even a class instance.

Making the Angular module configurable

To make the service configurable, it should be provided via a NgModule - a regular module might look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
import { NgModule } from '@angular/core';
import { CommonModule } from '@angular/common';
import { ReactiveFormsModule } from '@angular/forms';
// Module specific imports ...

@NgModule({
  imports: [CommonModule, ReactiveFormsModule],
  declarations: [FeatureFormComponent, FeatureViewComponent, FeaturePipe],
  exports: [FeatureFormComponent, FeatureViewComponent],
  provides: [ComponentService]
})
export class LibraryModule {}

To make this module accept a configuration we need to add a method that can return an instance of our module with custom providers. Providers are an important part of Angular’s dependency injection as these are the instance of what child components can inject - all will hopefully become clear through the code example.

First we will add a static forRoot method to the class. This method provides a factory function where we can pass a configuration in the application imports.

The factory returns an instance of that module with the provided configuration, and this passed into the items with Inject . The module can then be further used without the forRoot as long as it’s configured at the root of the application (see SkipSelf )

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
import { NgModule, ModuleWithProviders, Optional, SkipSelf } from '@angular/core';

// Here we can create a default object
const defaultConfig: ComponentServiceConfig = {
  storagePrefix: '',
  storageInterface: sessionStorage; // In our default value we will trust the global is there 
}

@NgModule({
  ...
})
export class LibraryModule {
  constructor(
    @Optional()
    @SkipSelf()
    parentModule: LibraryModule,
  ) {}
  static forRoot(config?: ComponentServiceConfig): ModuleWithProviders {
    return {
      ngModule: LibraryModule,
      providers: [
        {
          provide: COMPONENT_SERVICE_CONFIG,
          // Destructring allows the value of the config to be merged
          // this is a shallow operation so keep the objects simple
          useValue: { ...defaultConfig, ...config },
        },
      ],
    };
  }
}

Before we can use this in our application we also need to update the ComponentService itself to use this configuration. Here the Inject decorator will be used to tell the constructor that the ComponentServiceConfig this service needs will be available to inject via the COMPONENT_SERVICE_CONFIG token.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import { Inject } from '@angular/core';

export class ComponentService {
  constructor(@Inject(COMPONENT_SERVICE_CONFIG) private readonly config: ComponentServiceConfig) {}

  public load(key: string): string | undefined {
    return this.config.storageInterface.getItem(`${this.config.storagePrefix}-${key}`);
  }

  public save(key: string, value: string): void {
    this.config.storageInterface.setItem(`${this.config.storagePrefix}-${key}`, value);
  }
}

As the config is set with private in the constructor, TypeScript knows to set this as a property on the class. This is the same as writing the following code and provides a clean shorthand to set up properties.

1
2
3
4
5
6
7
export class ComponentService {
  private config: ComponentServiceConfig;

  constructor(config: ComponentServiceConfig) {
    this.config = config;
  }
}

Providing the features in the app

Now that we have our configurable module, the Angular application can be provided with a configuration when importing.

As the module now accepts any Storage type, we’ll instead provide localStorage for this application, along side the storagePrefix:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
@NgModule({
  declarations: [AppComponent],
  imports: [
    BrowserModule,
    RouterModule.forRoot([], routerConfig),
    LibraryModule.forRoot({
      storageInterface: localStorage,
      storagePrefix: 'my-new-app'
    })
  ],
  providers: [...environment.mockProviders],
  bootstrap: [AppComponent]
})
export class AppModule {}

Another example is we could have an NPM library that provides the same interface, such as localStorage-ponyfill , it could be used in your application - or you may want to provide it to a TestBed configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
// Using it in our application
import { createLocalStorage } from "localstorage-ponyfill";
const fakeLocalStorage = createLocalStorage({ mode : "memory" });

@NgModule({
  imports: [
    ...
    LibraryModule.forRoot({
      storageInterface: fakeLocalStorage,
      storagePrefix: 'my-new-app'
    })
    ...
  ]
})
export class AppModule {}

// Or writing some tests

TestBed.configureTestingModule({
  declarations: [TestHostComponent],
  providers: [
    {
      provide: BB_FORMS_CONTAINER_CONFIG,
      useValue: {
        storageStrategy: fakeLocalStorage,
      },
    },
  ]
});

With this technique, any kind of primitive or complex object, method or class can now be passed into your feature modules - and is a useful way to provide third-party libraries to your modules, configuring not just your services, but also components, pipes and stores.

Any class with a constructor (provided it’s in the same dependency tree) can use Inject with your exported token to access this configuration.

Hopefully you will also find this technique useful to improve the configurable and testable features of your module. Feel free to respond on twitter if you spot any typos or errors.