A POSTMORTEM OF ALX WEB STACK DEBUGGING 3

Issue Summary

At exactly 6:00 AM (WAT) Aug 8, 2023, ALX released a new web stack debugging project, a time to really show what it’s worth to be a Software Engineer — I mean what use is it to be called a Software Engineer if not to solve problems. Spinning up the container running the Apache web server, with curl I made a GET request to the localhost and got no response, passing the “is” flag I got “500 internal server error” a total downtime of our server 100% of request made to this server as from the time of downtime returned errors. The root cause of the server error was a typo in the WordPress configuration file, a require instruction for “class-wp-locale.php” was erroneously typed as “class-wp-locale.phpp”.

Timeline (all times West Africa Times)
6:00 AM: Release of ALX web debugging project
4:26 PM: Identified the downtime of the Server (start of project)
4:40 PM: Identified the configuration typo and fixed it
4:43 PM: Restarted the server, 100% of traffic back online

Root Cause

At 4:26 PM WAT I started the project and identified that the server is down using some dev tools I found out that the root cause of the server error was a typo in the WordPress configuration file, a require instruction for “class-wp-locale.php” was erroneously typed as “class-wp-locale.phpp”

Resolution and recovery

At 4:40 PM WAT, with some dev tools like tmux, ps, strace, and curl I was able to figure out that a request to the server tries to access a file “class-wp-locale.phpp” in the directory “/var/www/html/wp-includes/” but exits with ENOENT —open(“/var/www/html/wp-includes/class-wp-locale.phpp”, O_RDONLY) = -1 ENOENT (No such file or directory) — quickly examining the file “/var/www/html/wp-settings.php” I found out that a typo for the a require instruction for the file class-wp-locale.phpfileasclass-wp-locale.phpp and resolved the typo.

At 4:43 PM WAT, I restarted the server and we got back online 100%.

Corrective and Preventative Measures

I wrote a puppet file to automate, correct, and replace any phpp text in “/var/www/html/wp-settings.php” to php, this is a bit redundant but it’s will surfice for the current project at hand.

Our code made it to production due to lack of testing, it’s always the best approach to test before pushing to production.

Our server may likely have been down for days without the team knowing, due to lack of monitoring.
subsequently, we will need to implement a monitoring tool on our server with a tool like datadog and create alerts and triggers for the metrics perhaps using tools like pagerduty and developer(s) to be on-call to attend to any issue that may arise.

Sincerely,

Chukwuebuka Samuel Okoli
#ALX Software Engineering Student