Angular CLI flows. Big picture.

I don't know how about you but I got confused with a variety of tools that Angular CLI provides for some not straightforward Angular environment tasks. Builders, schematics, typescript transformers, custom tslint rules, AST — what are these all about, why do we need them and when do we have to use them? My brain was bleeding…

At last, I found a time to dig deeper and sort information about these tools. Let's review them one by one. This article uses Angular CLI 9.x codebase.

And we'll start with builders.

Builders

So what are builders in Angular? In Angular builders are used to do some routine tasks: build code, run lint, run unit tests, deploy code to host-provider. Here's what we can find in the docs:

A number of Angular CLI commands run a complex process on your code, such as linting, building, or testing. The commands use …builders, which apply another tools to accomplish the desired task.

Angular provides … builders that are used by the CLI for commands such as ng build, ng test, and ng lint. Default target configurations … can be found (and customized) in the "architect" section of the workspace configuration file, angular.json.

You can also extend and customize Angular by creating your own builders, which you can run using the ng run CLI command.

Let’s start with understanding what builders are used for and then explore how they are implemented.

If you run ng build command — Angular CLI actually runs the builder handler function (build in our case). Let’s go step by step and see what actually goes on behind the scenes.

Don't forget that your monorepo project can have a few applications, and in angular.json you specify builder for each specific project. And to start builder for a concrete project with Angular CLI you should add project name to the command, for example: ng build app1 (you can read more in my monorepo article here)

1. Read config in angular.json and find respective builder (projects->projectName->architect->build->builder)

"builder": "@angular-devkit/build-angular:browser", // original

OR

"builder": "@angular-builders/custom-webpack:browser", // custom

Here is code of build-angular:browser builder.

2. Create a builder instance and run it

export default createBuilder<json.JsonObject & BrowserBuilderSchema>(buildWebpackBrowser);

3. The builder runs its standard tasks:

• assertCompatibleAngularVersion
• buildBrowserWebpackConfigFromContext and runWebpack (webpack starts typescript compiler for your code)
• copyAssets
• generateBundleStats
• generate index.html
• Apply service worker if needed

and we get bundle files (index.html, css, js files in ./dist folder).

So what are builders used for?

Actually, they can be used for everything about your codebase: build, dev-server, run unit tests, run linter, etc:

Now you can assume what ng add command does — among many other things it adds new records to angular.json file (adding a new builder) — we will talk about ng add a bit later.

Let's run ng add @angular/fire in our project and check how angular.json is changed:

As you can see — a new deploy builder was added (so we can do ng deploy now for our project to upload bundled files to FireBase hosting).

Angular CLI standard builders

As you can see from picture above — standard Angular CLI builders are located in @angular-devkit package which contains build-angular collection.

Here you can find all builders like build, karma, browser, dev-server, etc and their implementations.

Custom builders

Also, you can create your own builder for custom purposes:

1. To add extra Webpack config options (custom-webpack builders by JeB Barabanov )
2. Concat bundled JS files (ngx-build-plus builder by Manfred Steyer)
3. Automate other routine tasks for you (configure and run source-map-explorer — by Santosh Yadav)

More to read

1. Angular CLI builders (official doc)
2. Angular CLI under the hood — builders demystified by JeB Barabanov
3. Custom Angular builders list page by Santosh Yadav

Conclusion

Builders are used to do some routine tasks: build code, run lint, run unit tests, deploy code to host-provider. Also, you can create your own builders to automate some operations and add some new possibilities: add Webpack configs, run scripts, concatenate bundled files, etc.

Schematics

Schematics transform your project: update files, install packages, add new component/modules/directives/etc files. Here's what we can find in the docs:

The Angular CLI uses schematics to apply transforms to a web-app project (modify or create project files).... Schematics are run by default by the commands ng generate, and ng add.

If you create a new version of your library that introduces potential breaking changes, you can provide an update schematic to enable the ng update command to automatically resolve any such changes in the project being updated (to do automatically changes in project code so code uses new API).

Also, this article states:

Schematics is a workflow tool for the modern web; it can apply transforms to your project, such as create a new component, or updating your code to fix breaking changes in a dependency. Or maybe you want to add a new configuration option or framework to an existing project

Well, it's still probably too vague. Let's make it more specific.

Do you remember how we added the possibility to deploy to FireBase hosting in the previous section with ng add @angular/fire command? We actually used schematics. What did this schematics do for us?

Here are these steps:

1. Installed packages @angular/fire, firebase, firebase-tools and so on package.json is also updated. @angular/fire contains a builder for deploying bundled code to FireBase.
2. Asked some specific questions while running (to specify options)
3. Updated angular.json — added deploy builder config. Now we can run ng deploy:

So schematics did all preparations work for us to start using deploy builder: installed packages and updated configs.

So how to run schematics

• ng new <appName> and ng generate <unitType> <unitName>— start respective schematic (using default schematics collection from @schematics/angular package).
  You can specify some default options for them in angular.json file — see more details here. Or you can set schematics options as a common line argument.
  You can override default schematics collection used by ng commands — modify project angular.json file (angular.json > cli > defaultCollection).
• ng add <packageName> — installs the package and then starts ng-add schematics from it (specified in its package.json).
• ng update <packageName> — installs a newer version of a package and then starts migration schematics from it (specified in its  package.json).
• You can also run schematics with schematics command  
  (watch more details in this video: A Schematic Odyssey by Kevin Schuchard & Brian Love)

Here is a nice review of angular.json structure: "Understanding the Angular CLI Workspace File" by @nitayneeman.

Where are standard Angular schematics located?

You may already know that you can generate a set of component files with the command ng generate component <some-component>. Where are these files templates are take from? You can find them in @schematics package:

So ng generate command just runs component schematics. Same with other schematics: directive, pipe, module, etc

Takeaway

• In Angular builders are used to do some routine tasks: build code, run lint, run unit tests, deploy code to host-provider.
• Schematics transform your project: update files, install packages, add new component/modules/directives/etc files.
• If you want to create your builder — deliver it with schematics (to be used with ng add) that update angular.json (add that builder into) and installed respective packages.
  A nice example of such a package is @angular/fire. It contains both builder (here and here) and schematics for ng add (here)

I will not stop on implementation details here. If you are interested — here is a list of recommended articles to start.

More to read

1. Generating code using schematics
2. Schematics — An Introduction
3. Effective automated scaffolding with Angular Schematics
4. Overriding Angular Schematics
5. ngx-deploy-starter — create your own deploy builder (and schematics)

Custom tslint/eslint rules for Angular

What is tslint for? It shows whether developers respect code style guide. Why do we need that? Style guide increases code readability (and so maintainability). And also there are rules that help you to prevent specific bugs (like rxjs-tslint-rules).

You can install and then update project's tslint.json file to start using newly installed rules. For example

npm install rxjs-tslint-rules --save-dev

//tslint.json

{  "extends": [    
     "rxjs-tslint-rules"  
   ],  
   "rules": {    
      "rxjs-add": { "severity": "error" },    
      "rxjs-no-unused-add": { "severity": "error" }  
   }
}

You can read more about rxjs-tslint-rules in its README file.

So when you run ng lint command :

1. Angular CLI checks for lint builder in angular.json file, create it and run it.
2. Builder starts tslint
3. tslint read tslint.json file to grab all the rules and then checks your code for compliance with the rules.

I will not dive in details HOW to create custom tslint rules here since it is out of the article scope, but you can read about it in proposed articles below.

Conclusion

• In Angular builders are used to do some routine tasks: build code, run lint, run unit tests, deploy code to host-provider.
• Schematics transform your project: update files, install packages, add new component/modules/directives/etc files.
• lint builder starts tslint which loads rules and then check your code to obey these rules — and you can create custom ones to expend for your code-style.
• You don't need Angular CLI to start tslint through lint builder — you can run it directly with tslint command (if it is installed globally) or npx tslint (if tslint is installed only locally in your project)

More to read

1. Custom TSLint rules with TSQuery
2. Migrating a TSLint Rule to ESLint
3. rxjs-tslint-rules
4. Writing custom TSLint rules from scratch
5. Custom TSLint rules — easier than you think

By the way, did you know that:

• TSLint is getting deprecated
• The angular-eslint project is a port from codelyzer
• The eslint-plugin-rxjs is a port from rxjs-tslint-rules

You can read more about it in the article Migrating a TSLint Rule to ESLint.

Custom Typescript Transformers

In the article Custom Typescript Transformers with Angular CLI David says the following:

The Angular CLI uses the AngularCompilerPlugin to transpile typescript. It is a webpack plugin that uses the typescript compiler together with various Typescript transformers to transpile the typescript to workable JS code for the browser.

Now let's show its place in a big picture:

• we start ng build
• Angular CLI(ng) finds in angular.json respective builder

• the builder starts webpack, and webpack uses AngularCompilerPlugin.
• AngularCompilerPlugin starts typescript compiler (to compile project .ts files) and provide specific transforms also to this compiler (you can read more about it here)
• And you can provide additional typescript transforms too (we are here)

What do standard AngularCompilerPlugin transforms do?

For example — transform "Inline resource"- takes component decorator templateUrl value (a filename), read the file, and put template prop instead with file content as a value.

A list of transformers is located here.

You can read about many of them in the nice article of Alexey Zuev "Do you know how Angular transforms your code?".

Why may we need to code our own custom transforms?

1. You want to apply your own syntax in Angular templates, and to make Angular understand it. ngx-template-streams uses that approach (video).
2. You want to grab some specific information from one file and modify another file during build process (Here is an article that describes such example)
3. You want to find all RxJS observables in your Angular project and automatically insert unsubscribe code (a nice article from Christian Janker about that).
4. etc

How do we provide our custom transformer so the typescript compiler can apply it?

AngularCompilerPlugin has a _transformers property where all Angular standard transformers are usually added to. So we have just to modify it and add also our own customer transformer. How?

There is a special ngx-build-plus:browser builder (it replaces standard builder for ng build command) from ngx-build-plus (by Manfred Steyer). This ngx-build-plus:browser that allows to modify internal webpack configuration in Angular projects (you remember, that ng build runs a builder that starts webpack to build our project, right?:)

Since builder has instance of webpack so you can provide your plugin for this builder and can get access to AngularCompilerPlugin instance and modify its _transformers prop by adding your custom transformer to the list.

Lets better describe it step by step:

1. We install ngx-build-plus and now when we start ng build command new builder is used — ngx-build-plus:browser (previously it was standard Angular CLI builder — @angular-devkit/build-angular:browser)
2. ngx-build-plus:browser can accept our plugin (it is not TS transformer, but webpack config transformer) where you can modify webpack config (Here is an example of such plugin).
3. So your webpack-modifier plugin gets access to AngularCompilerPlugin instance and modify its _transformers prop by adding your custom transformer to the list.
4. After ngx-build-plus:browser applied you webpack-config-changer plugin and get modifier webpack config — it runs webpack build or your project.
5. During the build process, webpack applies AngularCompilerPlugin transformers (and your transformer also among other transformers) — here is an example of such dummy transformer by David Kingma.

Phew ?

More to read

1. Having fun with Angular and Typescript Transformers
2. Hacking the Angular compiler with your own syntax [Video]
3. Custom Typescript Transformers with Angular CLI
4. Do you know how Angular transforms your code?
5. Converting TypeScript decorators into static code using tsquery, tstemplate and transforms!
6. Writing a Custom TypeScript AST Transformer
7. Using the Compiler API

Wrap up

Let's go through all parts once more (yes, you guessed right, I was a teacher many years ago, and my mom too ?):

• Builders are used to start some routine tasks: build code, run lint, run unit tests, deploy code to host-provider.
• Schematics transform your project: update files, install packages, add new component/modules/directives/etc files.
• lint builder starts tslint which loads rules and then check your code (typescript parser is used) to obey these rules — and you can create custom ones to expend for your code-style.
• To build angular code (which contains angular specific template constructs: *ngIf, [somePros], (click), etc) webpack uses AngularCompilerPlugin. It transforms Angular syntax converting it to something that the TypeScript compiler can understand. You can create your own template syntax (or typescript code syntax) for some purposes and make Angular (webpack) understand it too by providing custom transformer.
  Or you can use the transformer just to add something to code or modify it at build time.
  Transformers are applied only during build time.

AST Conclusion

Many of reviewed entities of Angular CLI (or related apps) uses Abstract Syntax Tree to do their work. Not to be embarrassed let's clarify that part too.

Typescript compiler parser can represent each .ts file as AST.

A linter scans an Abstract Syntax Tree (AST). Each specific lint-rule implementation is used to look for patterns in code (actually in AST).

Typescript transformers use file(s) AST also to look through and update files.

Schematics operates over project filesystem representation Tree (or Source)- you modify Tree and then these changes are applied to filesystem. (Tree is not typescript parser AST, read more here)

Builders don't use AST at all. BuilderHandler (builder implementation function) only gets input params and some architectural context (BuilderContext). BuilderContext just contains some project related information (ProjectMetadata, currentDirectory, etc).

Homework

Relax, just kidding you, no homework. Go feed your bear, play with your nuclear reactor, and don't forget to drink vodka (joking ?).

This article is a part of my own learning process. If you found some wrong or partially wrong statement — feel free to correct me in comments.

Let’s keep in touch on Twitter! Cheers!