Zendesk Headless Site Reliability & Performance
Project: Marketing Website Modernization
In 2022, on the heels of a successful content migration, my colleagues proposed another migration, to a “headless” CMS architecture. Specifically, we would build a new front-end application in React and NextJS. This NextJS app would consume the CMS’s content via the new REST API that I had built for the localization integration. This new microservices architecture would make it easier to scale content authoring and delivery and make global brand changes.
My front-end colleagues focused on migrating our design system to React and recoding our many templates using our new React components. Our infrastructure engineers extended our cloud architecture to support the new NextJS application and safely deploy our CMS in production for the first time.
There remained critical system design gaps as the two sides delegated what they didn’t own to each other. I stepped in to connect the dots, provide full-stack tech leadership, and ensure that we would continue to meet high standards for reliability, availability, and performance as we gradually launched the new stack.
It turned out that the firewall on our CMS API threw stray 503 errors during our builds. These weren’t just any pages either. The impacted page set consisted of our most critical pages: homepage, conversion pages, pricing pages, etc. Additionally, unbeknownst to us developers, the content team had developed a practice of unpublishing high-traffic and high value organic search pages in our CMS for 1-2 weeks at a time while they updated the content.
Since our legacy stack had incremental builds and page removals were manual, neither of these edge case behaviors had impacted the production site previously. On the new (more dynamic) stack, impacted pages would return 404 in production.
It was a classic case of Rumsfeld matrix "unknown unknowns".
To illustrate the severity of the problem, I staged a build (a.k.a. the “build from hell”) which passed all of our quality checks even though the bulk of the site 404ed.
My teammates had a mental model for the design of our new system that overcompensated for the flaws of the legacy system (manual removals of deprecated or renamed pages, over-coupling of content and code changes, use of passé languages like PHP); this had led to some blindness about the risks and design gaps in our new stack.
I could see the new system as it actually functioned more clearly.
I designed a tiered system that ensured 100% availability for our most critical pages with persistent fallback API responses cached asyncronously, solved the “build from hell” problem with a simple post-build test suite in Jest, and added asyncronous smoke tests to catch issues on the blog microsite (critical to our search rankings but too large to back up in our fallback cache).
I also worked with our cloud architect to load test the new stack. Our tests uncovered a critical need to increase our disk size, CPU, and memory for the NextJS app server prior to launch and validated that there were no issues with database availability.
Separately, the new stack was underperforming the legacy stack significantly on front-end performance benchmarks, both lab data like Google Lighthouse scores (especially Total Blocking Time) and field data like Datadog RUM and Web Vitals.
Our new React design system had been built on top of the Zendesk Garden product design system, which was great for inheriting Garden’s exceptional accessibility patterns but had led to client-side code bloat on our new React pages.
I met with our Technical SEO Manager and consulted with a Staff Engineer on the Onboarding Experience team who is an expert in React peformance and NextJS. I put together a front-end performance improvement plan which included tree-shaking work to drop unused code bundles from our front-end build, adding dynamic imports, template refactoring, image optimization at scale, and React and NextJS upgrades.
Another engineer and myself split the tasks over a 6-week period, gave the Technical SEO Manager weekly updates, and staged noindexed React versions of key pages for him on production. He pulled lab data that showed significant improvements, especially in reducing unused Javascript and Total Blocking Time. Upon launch, I monitored Datadog RUM Web Vitals scores and was able to identify additional fixes to move our remaining few yellow scores to green.