SEEK is Australia’s number one jobs site. With its online employment, educational, commercial and international businesses, SEEK has exposure to 2.5 billion people and over 20 percent of global GDP. The SEEK experience is seamless across desktop and mobile. SEEK Australia receives over 30 million visits per month, with over 50 percent of all visitors accessing it via mobile devices. In Australia, SEEK also has the SEEK Learning Business which connects people with education and training to advance their career and SEEK Commercial, where people can find businesses and franchises for sale.
Shane Lavelle, Senior DevOps Engineer at SEEK, is part of the team which supports and maintains the company’s primary Web and mobile sites in Australia (seek.com.au) and New Zealand (seek.co.nz). “If our site is not up and performing optimally, we're not generating revenue,” said Lavelle.
To keep pace with rapid growth, SEEK was adding resources which needed to get up to speed quickly and transitioning away from a traditional three-tier architecture to a service-oriented architecture.
With 12 projects running concurrently, and weekly software releases, SEEK needed to monitor application performance to ensure the technology continued to scale with business progress.
“At the time, our operations and engineering said, ‘Oh, yeah, we've got monitoring. We're fine, we're fine,’" recalled Lavelle. But SEEK was only monitoring its performance from an infrastructure perspective. Lavelle said the monitoring tools could identify high-level problems such as if the website was up, servers were running, CPU was within threshold and had disk space available. He added, “We weren't actually monitoring the real-time performance of the applications.” Though there were no major performance issues, smaller ones were lurking in the evolving architecture that could be missed as the team pressed forward, with the potential to cause future failures.
“We noticed the website was a little bit slower, it was not performing as it should, and we wanted to get an idea of how long page load times were, what the response times were, and where things were slowing down, or if there were bottlenecks,” said Lavelle. “That's why we started to investigate monitoring solutions.”
SEEK evaluated application performance management (APM) solutions in an effort, “To see our production architecture on one page effectively,” said Lavelle. “We also wanted to see where we were performing poorly, or where potential contention issues were.” After an initial discussion SEEK were introduced to Australian based AppDynamics elite partner ecetera.
The ecetera team worked alongside SEEK during their evaluation and Proof of Value demonstrating the capabilities of the AppDynamics solution.
The AppDynamics APM solution came out on top for its deep integration with multiple technology platforms. “We evaluated AppDynamics versus its competitor,” said Lavelle. “At the time, its competitor’s application allowed you to see the top layers of the .NET code, but you couldn't drill down. With AppDynamics, you're able to drill down in .NET and obtain code-level visibility.”
After an initial deployment supported by an ecetera consultant, SEEK were using AppDynamics APM as the starting point for determining a production issue, rather than relying on traditional methods, recalled Lavelle. Traditional methods required digging through database logs, guesswork and chasing down multiple teams — an inefficient process that often took as long as 10-12 hours.
“Whereas now, when an issue occurs, we can identify it within 15 minutes on AppDynamics,” said Lavelle. “We can see where it's coming from, the application that's causing it, and the line of code or the class or method call. Within another 15 minutes, the developers are able to identify what the actual issue is related to. They work on a fix, and within probably 45 minutes from the start, they've got the fix done, and we're already building the code. Within two hours, we've got it released to production?a fixed release.”
SEEK uses the dashboard capabilities in the AppDynamics APM to monitor each application project from a single view. “We have twelve streams of work and teams are working on parts of the application. Now, we can identify which part of the application the problem is coming from, so you know exactly which team to go through to work on the fix,” said Lavelle.
For example, when SEEK was transitioning to a new search engine, Lavelle set up a dashboard to monitor the old search engine to ensure it was no longer in use before being discontinued. “We could see there was a legacy application still sending off requests to the old search engine and the project team were unaware of it.” It turned out that a team within SEEK was relying on the data for end-of-month reporting. “Thankfully, we picked it up using AppDynamics and were able to change the code so it used the new search engine,” Lavelle added.
SEEK found that AppDynamics provided value right out of the box and has eliminated silos between several of their teams. “We don't use a lot of customization in the application because the standard metrics are serving the purpose," said Lavelle. "We can see each different application and how it's performing, which is helpful for the several teams using it."
With AppDynamics in place, Lavelle estimated a 30 percent reduction in manpower focused on troubleshooting, which freed resources to focus on more proactive application support. In addition, the data provided by AppDynamics helped justify the business case for infrastructure improvements and educate the company on the time and resources required for successful deployments.
“When estimating for projects, we were able to print off a snapshot from AppDynamics, and present to the business,” said Lavelle. The visual map helped project teams explain the complexity, justify the timeline, and convey the impact of a new project. “When the business can see our site on a diagram like that, they realize, ‘Oh, wait a second, our website is actually quite complex, it isn't just a front end with a database behind it’” Lavelle elaborated.
This increased application architecture awareness not only enlightened business groups within SEEK, but engineering as well. “Engineering, at the time, was then also able to start understanding the complexity of the website with regards to the applications,” said Lavelle. This led to more efficient software releases and better performance tracking. He added that when new code was being deployed, engineers could see the areas they needed to concentrate on for testing, and also benchmark performance before and after the release. “They could go back 3, 6 or 12 months to see what it was like back then, identify the trend, and how it has changed and what has been introduced over that period of time that's effected these changes,” said Lavelle.
The application visibility and benchmarking capabilities of AppDynamics supported collaboration between development and operations and SEEK’s move to a DevOps model. “With AppDynamics, we were able to look at both ends of it. We were able to see the application and how it performed, and we were also able to see the machine agents and the infrastructure side of things. We then moved to the DevOps model so release engineers were responsible for their own assigned project, and deploying it all the way to production.”
“Things started to run a lot smoother, we began to have fewer issues in production, and we could see this using AppDynamics,” said Lavelle.
The teams at SEEK have become accustomed to using AppDynamics from inception, to design, through implementation. “More and more, teams are starting to use it as a tool for planning and designing when they're making their decisions about the best way forward for an idea or a new product,” concluded Lavelle.