What is Lighthouse
Lighthouse CI (LHCI) is an open-source, automated tool for improving the quality of web pages. Built by Google, LHCI makes it easier to test, archive, and browse reports on SEO, performance, and accessibility. Developers and site owners use these reports to take action against and improve their applications.
- Run LHCI reports alongside every PR.
- Prevent regressions in accessibility, SEO, offline support, and performance best practices.
- Track performance metrics and LHCI scores over time.
- Set and keep performance budgets on scripts and images.
- Run LHCI many times to reduce variance.
- Compare two versions of your site to find improvements and regressions of individual resources.
Where should I run my Lighthouse tests
When testing web applications there are multiple areas to consider, functional vs non-functional, code standards, visual regression, performance, SEO, accessibility, and more. Taking a step back, we first need to determine which test should run in which environment and scenario.
In the CI/CD build, DEV or PROD
Testing in the CI/CD build
Whether you are using Jenkins, Travis, CircleCI, Bitbucket, Gitlab or Acquia’s Code Studio, our Drupal-optimized CI/CD and code collaboration solution built around Gitlab. If we look at CI/CD, focused on compiling and testing the code then there are a AutoDevOps you can run in the CI from code standards, PHPcompatibility, Semgrep, SAST to unit testing, these are all good examples of the kinds of tests you would run in the CI build process.
Using the build process to focus on testing the code will ensure your team that all code on its way to PROD has been tested and validated. Another consideration is the time it takes for each build to run. Taking into account that you might have a team of developers pushing code, how much total time do you want between pushing and the availability to deploy?
As mentioned above, the build process can play a valuable piece of your DevOps puzzle. Further in this article, I’ll explain how to use CI/CD to set up, and archive LHCI tests to run on different environments.
Testing in a temporary environment or DEV server
At this point, the code has already been deployed and is on one of the servers, DEV, Non-PROD or PROD. Having the code tested prior to this stage ensures that your team is testing a validated artifact of the application. This is a good time to start testing other aspects of the application, like Accessibility and SEO.
Running your LHCI test here makes sense, as you have the Database, Files and Configuration ready to go. That and the fact that it's an environment with known resources to benchmark against.
At Acquia, we have deployed CDEs, or Continuous Development Environments that are integrated into our Code Studio product to spin up new CDE environments on any new merge requests. This existing process gives us some interesting tools to build our LHCI testing into.
When thinking about performance testing, some customers choose to rerun these tests in PROD to benchmark against. When doing this, consider that the performance tests will not be comparable, as the Non-PROD environments usually do not have the same allocated resources as PROD environments.
Are these real tests if I’m not running them in PROD?
What is real? Does your UAT or TEST server have the same allocated resources as PROD? Can you observe the traffic and load on the server while maintaining efficient page load speeds.
There are things to think about when it comes to the different kinds of tests, benchmarking, measuring, and iterative. For all these examples, the goal is to improve and mitigate risk. If you keep that in mind, you can develop your own testing procedures that will lead to a successful DevOps process.
While Lighthouse’s primary focus is performance, it can assess other things like:
- Accessibility: Recommendations on making content more accessible. This ensures that the page is optimized for screen readers, all elements have labels, or the site is browsable with the keyboard.
- Best practices: Scans the page issues not following best practices and can improve performance and security.
- SEO: Runs different checks to ensure that the page is SEO-friendly