Slovenčina Have you ever wondered how news leaders measure the impact of their articles? So did we, and we asked eight top media professionals and data analysts to lift the curtain on their evaluation methods. Our question was simple: 'How do you evaluate the success of an article?' Read what unique metrics they lean on to define article success.
Game changer: conversions over pageviews
Dávid Tvrdoň, Subscription Growth and Retention Specialist at Denník SME (Slovakia)
“A few years ago our primary focus used to be purely on looking at the number of page views of articles, which is undeniably still a very relevant metric. However, nowadays our newsroom has evolved and looks at subscription conversions attributed to a single article. While the editorial team has access to many other metrics, such as time spent on the article, which we like to look at globally, subscription conversions have become a key metric we prioritize in our evaluation process.”
Tomáš Bella, Chief Digital Officer at Denník N (Slovakia)
“Our key metric for assessing article success is the number of new conversions per article, basically representing the number of readers for whom the article was the last trigger to motivate them to buy a new subscription. While this metric is straightforward and universally understandable, it has its shortcomings. Mainly, it doesn't take retention into account, only new subscriptions. Nevertheless, it is so simple and easy to understand for everyone that we use it as the key indicator in the newsroom. Our journalists see it displayed on live dashboards, they get daily overviews of all articles published and weekly summaries of their own articles. Additionally, every month they get a list of the best converting articles, and journalist bonuses are partially based on this list.
In addition to new conversions, we also analyze various article metrics. This includes:
- conversion rate of the article (pageviews vs. number of conversions)
- the number of new registrations triggered by the article (through registration paywalls)
- the number of unique browsers that visit the article
- ratio of visits by subscribers versus non-subscribers
By examining these comprehensive metrics, we gain a deeper understanding of article performance and its impact on audience engagement and subscriptions.”
Dennik N uses the module called BEAM of the Reader Engagement and Monetization Platform. Read how REMP can help you grow your subscriber revenue.
A snapshot of the dashboard provided by the BEAM module of REMP that the newsroom relies on every day for key metrics.
Lars K Jensen, Team Lead Audience at Berlingske Media (Denmark)
“We look at multiple parameters: we take into account both quantitative measures, such as views, as well as qualitative measures that assess the quality of consumption. Additionally, we prioritize understanding user needs and preferences. Currently, our primary focus is on conversions, which involve people purchasing subscriptions or signing up for user profiles. However, we are also committed to delving deeper into understanding our users and subscribers on a more granular level. We are good at “keeping” them and ensuring they feel welcomed.
To achieve a well-rounded assessment, I would suggest other publishers to think of a way where you can semi-qualitatively measure and assess your journalistic output on a quantitative level through the eyes and experiences of your readers. I have been doing this as a part of my daily work for the past two years, and I wish we had started it earlier, as it has proven invaluable."
The subscriber shuffle: acquire, keep, repeat
Lukas Gorog, Data Strategist at Die Presse, Founder at Predictea (Austria)
"When evaluating the success of an article at our organization, we employ two distinct audience approaches: subscriber acquisition and audience retention. For subscriber acquisition, we rely on a combination of pageviews and engagement metrics. An article is deemed successful when it performs strongly in both these areas. Engagement metrics adhere to a precise formula that includes user events, like Newsletter subscriptions, Time on page, Author subscriptions, and number of loyal readers per article. This score ranges between 0 and 1.
To quickly assess the effectiveness of our new subscriber acquisition tactics, we conduct various A/B tests. For instance, we A/B test the performance of our recommendation algorithms, the impact of various page layouts, and the effectiveness of different paywall positions on our website.
In terms of audience retention, we analyze Page Views and Engagement metrics, focusing specifically on our loyal subscribers. This approach allows us to evaluate the quality of our product and its alignment with our long-term strategy. By analyzing these metrics, we gain valuable insights into how articles impact audience retention, empowering us to make informed decisions about our content strategy."
[If you want to go beyond pageviews too and measure the success of your content with the metrics like newsletter subscriptions, time on page, author subscriptions, and number of loyal readers per article, reach out to our FatChilli team at [email protected] ]
Snapshot of a Traffic Source report from a dashboard developed by FatChilli
Emanuele Porfiri, Senior Data Analyst at FT Strategies (UK)
“Though obvious, we believe there isn’t a one-size-fits-all solution when it comes to measuring content performance (it depends on the organisation and content’s goals, as well as the data available). At the FT, we evaluate the performance of an article based on the volume and quality of engagement it generates. Whilst volume is perhaps a straightforward metric, quality of engagement is a trickier one to define and we opted for a metric we call ‘Quality reads’, that based on a set of page views and article’s indicators, estimates the percentage of page views where the reader has read at least half of the article.
Ultimately, as there are endless ways in which to measure performance, at FT Strategies we recommend testing different versions (if the data is available) and validating them against a desired user action (e.g. registration, subscription, retention…). At the FT, we can see that subscribers with higher quality engagement are less likely to churn, and that’s exactly one of the behaviours we want our content to generate.”
Delving into deep reader engagement metrics
Andrey Boborykin, Executive Director at Ukrainska Pravda (Ukraine)
“While we start with the fundamental metric of pageviews to gauge initial reach and interest, we recognize that true engagement goes beyond mere clicks. To gain a more nuanced understanding, we delve into scroll depth and engaged reading time. By analyzing how far down the page readers scroll, we can infer the level of their engagement. Furthermore, we measure the amount of time readers actively interact with the article, which provides valuable insights into their engagement. These metrics are then evaluated against our internal benchmarks, allowing us to assess not only the absolute numbers but also the relative success of each article. Such comprehensive analysis helps us refine our content strategy and ensures that we consistently deliver relevant and engaging content to our readers.”
Joao Rodrigues, Data Specialist at Observador (Portugal)
"Our approach to measuring the success of an article has changed over time. Through numerous iterations and collaborative efforts between the editorial and subscriptions teams, we have arrived at a formula that is now widely accepted in our newsroom. We believe that a combination of three key factors is crucial in defining success: reach (pageviews), engagement (time and scroll depth), and conversions (number of subscriptions resulting from a specific article). Evaluating these three factors together helps us achieve a more comprehensive assessment of an article's success and continuously refine our content strategy, ensuring that we deliver impactful and engaging articles to our audience.
We circulate these metrics to our journalists, editors, and directors daily. Directors use them to know what content to highlight on the homepage and which articles to pick in their daily and weekly newsletters. Editors analyze these metrics to understand which articles (and their respective subjects) have performed well in recent days. With these insights, they make informed decisions on what to write (or not to write) next. Journalists use these metrics, offering their unique perspectives on why they believe an article performed better or worse.”
Measuring the immeasurable? The challenges of quality and impact assessment
Greg Piechota, Researcher-In-Residence at International News Media Association
"Evaluating the success of journalism is a subject of wide debate. Although many agree that the purpose of journalism is to make an impact, they agree less on how to measure it. The work of journalists can be easily measured by activities and outputs, but evaluating output quality and impact is trickier. It includes checks for adherence to the craft standards, metrics of user engagement, and recognition by peers, e.g., citations and awards. The impact of journalism on individuals, society, and institutions is the hardest to quantify.
User engagement is often measured with usage metrics, with the most common showing popularity - how many people visit the website or click on the links. Then there are metrics showing how engaged readers are with a website, how many articles they read, or how much time they spend each time they visit. One often measures whether users read or add comments, share articles with others.
Metrics of loyalty are also crucial. For example, how often users visit the website, how many were present on a particular day or in a given week or month, how much time has passed between visits. In the news subscription context, a common metric is lifetime value (LTV) which measures the expected revenue or profit over the lifetime of the relationship - from the day someone subscribes to the predicted day of cancellation.
Engagement is highly individual in digital media use. People have different ways of engaging and engagement varies across website types and pages. It also differs based on the context of a visit and user intent. People behave differently when they just want to kill time and find out what's going on in the world, and differently when they are looking for specific information. Understanding these diverse behaviors is essential, otherwise comparing them will be difficult.
Success is more than just numbers, right?
As we've discovered, the magic of a successful article goes beyond numbers - it ignites curiosity, sparks conversations, and provokes thought. In the quest for success, one shouldn't be confined by data alone. Rather, go and seek out the intangible connections that resonate deeply with your readers. It's these elements of empathy, authenticity, and a genuine passion to enlighten that truly set an article apart. After all, the road to success isn't just paved with data - it's built on the foundation of genuine human connections and lasting impact.
If you're ready to take your article analytics to the next level, reach out to us at [email protected] and learn how FatChilli can support you in making data-informed decisions. Or, to stay updated on the latest trends and insights in digital journalism, subscribe to our newsletter
Note: Cover image generated by Midjourney AI
