Introduction

Improving a product is driven by learning about what potential customers like, dislike and need from it. With online products, this process can be rapidly expedited by capturing and analysing important customer-centric metrics such as the searches they make and the paths they take through your website.

Using measurements to drive rapid product improvement is at the heart of The Lean Startup. A book which continues to inspire the product-development process of many businesses – from day-old startups right up to Netflix and the US government. There’s a good chance that at least one of your competitors are using lean concepts and customer-centric metrics to gain competitive advantage.

Building metrics capturing and analysis tools to support the development of your product can be a big development effort in itself. In this post you’ll see some of the tools we use at Artirix to provide an out-of-the-box solution for our clients to immediately be able collect and analyse their important business-metrics

Whether you want to build your own solution or are considering using ours, you’re more than welcome to ask us any questions about metrics in general or how our system can help improve the rate of development and quality of your product.

Real-time trends and dashboards

For some businesses, short-term decisions can have a significant impact on revenue generation. Think about financial traders, for instance. They make split-second decisions about buying and selling shares based on the latest stock market trends.

Real-time trends aren’t just for finance, though. On a few our projects, we see profit margins being adjusted on a minute-by-minute basis according to which products and services are selling the best. In fact, weusage of real-time trends on projects ranging from travel to real-estate.

TOOLS FOR VISUALIZING REAL-TIME TRENDS

We’ve been overwhelmed by the positive reaction from all levels of technical and non-technical staff for our preferred data visualisation tool – Kibana, as shown in the next image.

Kibana has been at heart of a number of product improvements and business insights across our project. On one project, we worked closely with the business to understand their pain points. They told us that users seeing no search results is a big loss of revenue. Understanding their needs, we helped them set up a Kibana dashboard showing failed searches grouped by different parameters such as destination, departure airport and date ranges.

Using the dashboard, the business were able to identify hotspots of failed searches and immediately re-create the experience there users were seeing. This led them to identify their supplier as being at fault for not meeting critical SLAs. At peak times, some suppliers were timing out and returning zero results, even for valid searches.

One aspect of Kibana that was incredibly useful in this case was the ability to inspect individual data points enabling those exact user journeys to be re-created. The image below shows an example of how easy it is to inspect individual data points with Kibana.

Kibana is a very usable tool that makes it easy for non-technical people to create fully customised dashboards of their important business metrics. If you would like to learn more, feel free to contact us.

Longer-term trends

You’ll often want to understand how your metrics change over a longer period of time. Businesses usually want to understand how conversions and other metrics compare to yesterday, last week, last month and last year.

For one of our travel-related projects, understanding failed-searches was again an important aspect. By visualising trends in Kibana, the business were able to identify holes in their catalogue where there was a significant user demand. Potential customers wanted holidays in Greece and other locations during the month of February, but their searches were returning nothing.

The business had no availability in their catalogue for some regions, including Greece, during the month of February because they assumed there was little demand. But a Kibana dashboard showing popular searches, returning no results, grouped by destination, clearly indicated that Greece was as popular in February as some other months when they did have availability.

By taking notice of their metrics presented as a Kibana dashboard, the business were able to improve their product by adding additional availability for destinations that customers wanted but was previously unavailable.

There are endless types of metric that can be measured as trends over periods of time including performance, conversions, diversity of search results, user journey. Feel free to [contact us] if you’d like to learn about what we’ve done on some of our projects or could do for your projects.

Detailed Ad-hoc Analysis

When you want to answer specific questions about your product, you need to be able ask specific questions of your data. For example: “how many visitors to our website clicked through to an offer details page for products marked up as special?” or “what did users who saw zero search results do next – did they make more searches and convert, or did they leave the site, and thus we lost their potential custom?”.

Questions like these can be precisely answered by making ad-hoc queries of your data. On some of our projects, we’ve been able to improve the product by following the journey of visitors who didn’t convert to understand why they didn’t convert. Often we find that the performance was slow or we didn’t make it easy enough to find the products and services they were looking for.

One one project, we improved the user experience based on what our metrics told us. We could see that a lot of users attempted to search for products, but then suddenly left the site. The problem was one of the options on the search bar. It was almost hidden and causing a validation error in the browser. Lots of users found this very annoying and decided to leave the site rather than correct the problem (simply by selecting an option from a drop down) so they could carry out their search.

By improving the visibility of that one small drop down list, the bounce-rate was improved massively and more users were getting through to the search results page. This also coincided with an increase in conversions and revenue.

Tools for detailed ad-hoc analysis

Reporting fine-grained user-behaviour metrics requires massive amounts of data storage. For this need traditional SQL databases are not a great fit. Unfortunately a lot of businesses understand SQL very well and has been a specific demand of theirs.

To get the best of both worlds we store huge volumes of data on Amazon S3 for minimal costs ($1 per-GB per-month). And we use a querying tool that allows the data on S3 to be queried using full SQL syntax. We have had massively-positive reactions to this from the business on our projects; both regarding the cost-effectiveness of this solution and the full-features provided by SQL that are still available.

Below is an example from one our project using the tool to find the holiday destinations which users were most searching for but seeing zero results.

Metrics Consolidation Enables Deeper Insights

Some of the examples we’ve spoken about in this post have shown an importance in being able to pull data together from different sources. You saw that pulling in front-end user-behaviour metrics (right down to mouse clicks), can be combined with back-end metrics like supplier performance to give important insights that drive product development.

On some of our projects, we have been very limited when data lives in multiple places. Some metrics are tracked in google analytics, while front-end mouse-clicks are reported to one product, and back-end API monitoring are sent to yet another. This can limit the insights that you are able to glean.

You should still consider using a variety of products, but you should also try to have all of your metrics available in a single place as well. This is important if you want to be able to maximise the opportunity to learn about ways to improve your product.

At Artirix, we have ready-to-go solution that allows you to have all your data stored in one place and made fully accessible to all of analysis tools we use such as Kibana and our SQL querying tool. This allows you to slice and dice your data in new novel ways without any restrictions.

Here’s how our solution works:

As you can see, we can take a big-picture understanding of the data coming into the system, how each part of the system behaves and how users are using the system. We can even pull in data from twitter and other social media sites.

With all this information stored in one place, you can query and combine your data in any way you like to gain important business insights that drive the rapid improvement of your product in a measured way.

Collaboration Increases Possibilities

On software projects there’s always a gap between what the business thinks is possible from the technology and the development team’s understanding of the businesses needs. By spending time together, the developers can learn about the domain and business can understand what can be produced by the technology.

On most of our projects, we see that as developers learn more about the problem domain, they are able to contribute by making suggestions about metrics to capture. At Artirix, our developers are addicted to data, and they often present business cases to the business about how their product could be improved.

But what we also see through close relationships between the business and development is that the business understand the technologies better. With a better understanding of the technology, the business often come to the development team with specific analysis requirements that support creativity and product development.

Closing Thoughts

The modern business needs to use metrics as the basis for developing their online products – simply to keep pace with their competitors. This can be a costly and lengthy process for some development teams who are starting from a fresh slate.

In this post we’ve shown you some of the tools that can be used to get going. The tools we’ve shown are actually Artirix components used in the services we've built so you too can use them to start capturing and reporting on the metrics needed to rapidly develop your product.

Whether you decide to build your own solution, or consider using Artirix, good luck.


Ready to get started?

OR EXPERIENCE OUR WORK OR LEARN ABOUT OUR APPROACH.