How-to guide: Building a modern customer loyalty engine

In a previous article, we reviewed the main attributes of loyalty programs including customer profiles, points systems, and rewards types. We looked at what types of business inputs are required to design a loyalty program and which business rules and strategies need to be clearly defined. In this article, we provide a comprehensive guide on what features a modern loyalty engine should have.

It’s an important task to prepare input data, design output data, and make sure all upstream and downstreams systems are in sync. There are multiple activities that need to be planned accordingly and carefully checked. We will discuss every step of these processes as well as talk about how to build a loyalty system, what options are available for retailers, and the main pros and cons of each option. We’ll also present a sample loyalty engine architecture and discuss how to most effectively use each component and module.

To be able to build an efficient loyalty point-based program for all customer profiles, a retailer needs to have access to the right kinds of data including transaction history, enrollment details, insights, customer profile details, and most used payment methods.

Calculation of the loyalty points is mainly based on the detailed transaction history of a particular customer. The transaction history should include not only order amounts but also requires an understanding of the tax amounts, shipping costs, and any applied discounts or rewards. This allows you to be able to accurately add points to a customer account or calculate any returns in the future. The main source of such information will be the CRM and/or eCommerce platform, depending on the retailer ecosystem in place.

To predict what factors will be the most important or valuable for a customer, retailers should collect and manage customer information and insights. This includes multiple attributes such as age, habits, location, job, income, and birthday. The main sources of this type of information are eCommerce platforms and POS, where this information can be collected from channels like the web, mobile web, mobile applications, and in-store.

The credit card provider can also be an important source of information if a retailer issues branded credit cards. Credit card providers can share information about transactions placed with retailer branded cards, so a retailer can analyze other purchases and customer needs and additionally reward customers for using a branded credit card.

Another useful source of information is customer subscriptions and activities. A retailer can track if a customer has subscribed for email lists and/or SMS subscriptions and what exactly the customer has subscribed for.

In the previous article we discussed the main options available for building a loyalty solution and their advantages and disadvantages. Essentially there are two paths that can be taken; building in-house, or using a 3rd party loyalty platform. Research shows that large companies tend to opt for building their own in-house solutions. There are a variety of reasons for this including having greater control over data and better meeting the needs of the marketing team. Small and medium sized companies more frequently opt for a 3rd party loyalty software, as these tend to be available at lower upfront cost and in a shorter time frame.

But no matter what implementation option you choose, there are a number of challenges you should be expected to face. So are you ready to find out about them, including the most common problems you’ll encounter and what is likely to cause the most difficulties?

That’s right – data preparation. Once a retailer has decided on the type of loyalty program they need, established the business rules, target audience, promotions, etc., they must then understand their data requirements. This is necessary to be able to tally points for tier calculations and knowing which products are required for setting up a promotion.

Technically we can group all required data into the following groups:

• Customer data – this was discussed in detail in Article 2
• Transaction data – you need to know what your customers are buying, how often, transaction amounts, payment type (credit card, paypal, gift cards, etc), and any coupons applied to the transaction
• Catalog data – this type of data is needed for multiple reasons. Firstly, you need to build a vocabulary of products to be able to easily understand what your customers are  buying. Secondly, for promotion configuration if it’s based on the product, such as – “Buy two pairs of skinny jeans, get a sport’s bag for free”
• Store data or channels – by gathering statistics from physical stores, its possible to understand key information such as which channel is most used by customers, in-store or online

Great! Now we know the kind of data needed to run a loyalty program. The next challenge here is how to collect that data by using historical data migration. This step is the most important and complicated. It’s so important that I would even advise managing it as a separate track or project.

Just imagine every organization has multiple internal and external systems with duplicate data. This means you need rules governing how to define the right records, and mapping rules to determine how values should be calculated from multiple sources. A great example here is “order total”. Let’s say you have a requirement to calculate points per transaction, which requires being able to define the order total. What is the best source for this data? Is it up-to-date? Is it full? Is a transaction presented as a single record? What do you do if there was a return requested for this transaction? What’s the effect of having item price or cart prices saved in one location and information about taxes, shipping etc in another place?

The above are just a short list of questions you need to be able to answer. These types of questions should be asked for each and every attribute of your loyalty program. In the case of a 3rd party loyalty program solution, you need to clearly understand its schema and the entities relationship and provide the expected data. You also need to undertake detailed testing to prove and confirm that you’re sending the right data to the 3rd party vendor.

So is that it, are you now ready to go? Hmmm… no. While the most painful part has been done, there’s now a new challenge – data synchronization. In the previous step, we discussed defining the list of data sources and the required attributes for each source. Now let’s consider the situation where one source is updated daily, another one weekly, and a third one monthly. How can you keep all the data in sync?

With the modern world changing so rapidly, ideally everything can be updated in real time or near real time. But that’s not always possible. So you need to think about the frequencies needed for updating transactional data and customer data, and what the right mapping attributes between systems will be. It’s also important here to consider the communication method between systems, will it be a file transfer, API call, or messages?

Now, what’s next? Data quality. To properly manage all collected and calculated data, we need to think about quality standards. It’s important to have cleaned data without any duplicates or missing mandatory attributes, and the data should be aligned for all customers. Every retailer needs to determine exactly what’s needed in each customer record and which attributes are mandatory and which are optional.

When thinking about optional data, try and consider what types of information while not essential now could become crucial in the future. You also need to think about the rules required to populate this data and ensure that these rules are carefully reviewed as one missed rule that generates bad data can destroy an entire loyalty program.

One more item to consider is time to store the data. It’s clear you can’t save all data forever. You need to set up a time frame for every piece of data. So for example transaction data might be necessary to save for 3 years but rewards data or customer data may only be required to be stored for one year.

It’s also vital that retailers track their success by using analytics/reports. Be sure you document all goals, ideas, and formulas to calculate each and every attribute. It will save a lot of time if a system is destroyed or you want to make some adjustments.

Now let’s think about all these problems for the case of a retailer choosing a 3rd party loyalty program solution to build their loyalty program. Expect the retailer to double or triple the complexity of the problems outlined above. Start from the very first step: data schema. Just imagine you completed all the tasks above, you fulfilled all the required attributes and now your vendor is saying to you, “You need to follow my data schema”. It means you need to repeat all the steps from scratch. How does that sound?
So you’ve now set up a data contract and started to provide data to a vendor. The next surprise is data governance. You face the prospect of having little to no control over customer data. Every time you need any piece of data – it’ll be a legal request and a vendor can deny it. And keep in mind that if a vendor decides to or is forced to close their business, you’ll lose everything.

Moving forward, all business logic like points calculation or rewards assignment will become a black box for you. This is because you won’t have direct access to code and there’ll be no ability to verify or refresh any particular rule. Most 3rd parties support a predefined set of business rules, so adding a new rule out of the box will lead to a change request. As a loyalty program is a living system that needs to be updated frequently, in a couple of years it will be problematic to conduct an audit or verify data.

To protect against customer fraud, points and rewards thresholds need to be established. For example, “a customer can make as many points purchases per day as they want but the maximum number of points that can be earned in a single day is limited to 2000 points” or “a customer will be rewarded only for the first 5 purchases per day”. If it is detected that a customer has received more points than the threshold limit, then the customer’s loyalty account will be suspended until an investigation has been carried out.

Imagine you’ve decided to add a new feature that wasn’t present in the vendor’s solution. You send a request to have the feature added, the vendor reviews it and you start to clarify requirements, prepare the SOW, approve it, confirm the budget, and schedule development. Even adding a small feature can take a large amount of time.

It’s important to consider the importance of adding a new feature and what effect it will have. The most commonly requested and valuable features requested from a marketing perspective include:

• Adding a new promotion template – most 3rd parties support only predefined lists of promotion templates and even a slight change of a promotion condition can lead to a change request.
• Adding a new business rule – a loyalty program is a set of business rules. So naturally the business will want to make changes from time to time to follow and meet customer requirements and needs. But again if it’s not in the box, it’ll be a change request that will likely result in a fairly long wait to have it implemented.
• Adding or changing a loyalty program currency – businesses want to be more flexible to meet a wider range of customers so a common request for a scaling business is to use a different currency for an expansion to serve customers in a new region.

The next problem is customer support. A good example to discuss here is the situation where you have determined there is a data issue, but what is the process to resolve it? You submit the issue, someone from the vendor side starts working on it in accordance with their sprint plan and priorities, but meanwhile the data issue is becoming a major problem for you.

In our experience, we have faced situations where the API responsible for customer rewards in real time was not responding for a long time and as a result the end customer was not able to check and apply their rewards. And the CSR agent wasn’t able to help as the issue wasn’t visible at their end. In another example, the CSR agent reviewed a customer profile but there was missing data or points that weren’t being properly deducted for some reason. Again you need to call 3rd party customer support and be prepared to wait. Meanwhile the mistake can be affecting multiple customers and it will be difficult to clean the data even after the problem is resolved.

At the end of the problem list we should talk about subscription fees. To use a vendor solution you need to pay a fee, nothing is free. It depends on the particular vendor how that fee is determined. It can be a flat rate, can depend on the quantity of customer accounts to be supported, or be a percentage of margin. You can also expect that the fee will grow larger as the business grows. The risk for larger companies is that over time the vendor fee can grow to be very significant but to move away their loyalty solution can be very difficult. This often leads to the situation where one day you suddenly realize how much you’ve paid to simply create a giant headache for yourself.

Every retailer needs to understand what the main components of a modern loyalty program are. This includes how it is used and what its main functions are. Every business is unique and requires different needs. The most important of them are to run a loyalty program with a pre-configured list of business rules, to set promotions linked with a loyalty program, to manage a customer account, and to properly assign rewards to customers.

Let’s review what components are needed to build a successful, reliable, and legal loyalty program.

The Loyalty Service is designed to calculate the customer tiers, points (any intermediate currency), and rewards based on incoming transactions and rules set up by the Program and Promotion Portals.

When a loyalty customer makes a purchase at a retailer via any channel (like online, in-store, mob app, etc), points will be awarded based on the qualifying purchase amount excluding discounts, taxes, rewards, etc if such rules have been set up.

The following types of points can be defined:

• Base points – points awarded for every $1 spend (or any configured amount to be spent)

The source of base points can be from transactions made by a customer or points added by customer support for some reason like a missed amount or due to performing a correction. Depending on the type of loyalty program that a retailer wants to build, base points can be different for a particular tier. The qualifying purchase amount for the calculation of base points excludes the following values: taxes, shipping, gift with purchases, discounts or any other rules set up by a retailer.

• Bonus points – points awarded for any promo that a customer used in their transaction

Bonus points might be rewarded due to retailer credit card transactions or promotions.

• Rewarded points – points converted into rewards
• Expired points – points that are moved to an expired state due to customer inactivity

Based on the configured rules for every type of points possible, a required lifecycle can be set up as: pending, available, or total points.

The most complicated parts in the points calculation process are building the right return policy as well as setting the right timing. The main rules that should be implemented for the return policy are:

• If a customer doesn’t have any points to be deducted, then the customer balance should remain as 0 points and should never be negative
• If a customer made a purchase and also received promotion points, then promotion points usually are not deducted, only base points should be reduced
• Shipping cost, taxes, duties are not used in calculations

For the loyalty service to work properly, it needs to be fed the right data and the attributes mapping must be done correctly. With an in-house solution this step is much easier. This is  because internally you have a much better understanding of what kind of data is stored and available, how it can be used, and in what ways it can be easily modified.

Working with 3rd party loyalty program solutions, you will face the same set of problems but to a greater degree. This is because you also need to clearly understand the schema of the 3rd party loyalty program solution and then match the data you have to it.

The customer service portal (CSR) provides the ability for support agents to help loyalty customers with any questions related to the loyalty program. Skillful customer service is very important but is no simple task. But it can be helped by the customer service portal simplifying the process of getting information about the status of loyalty accounts and responding quickly to customer requests.

CSR agents are able to perform the following actions in the loyalty customer service portal:

• Search for members by Loyalty Number, Customer Last/First Name, Customer Zip/Postal Code, Customer Email Address, or Customer Mobile Number
• View customer account activity
• View customer account details
• Manage customer account status (Freeze, Unfreeze, Terminate)
• View Rewards History and issue Rewards Appeasements
• View Tier History and modify Customer Tier
• Add points using Point Adjustment/Point Transfer
• Send requests regarding missing purchases
• Add customer service notes
  

Again comparing in-house and 3rd party solutions, once a CSR agent identifies an issue, it can be fixed by the development team immediately. In the case of a 3rd party provider, there is far less control over the timeframe over which the issue will be resolved once it’s been submitted and the vendor prioritizes it.

The Gamified promotion configuration portal provides the ability to create/update/view/stop a promotion. A retailer can set up a required promotion life cycle to fit their business needs.

Standard promotion attributes supported during a promo creation process are: Promo Name, Status, Start Date, End Date, Type of Promotion, Return Strategy, Audience, Currency, and Payment Types. The list of promo attributes is configurable and a retailer can add any attributes that the marketing team needs.

A basic promo lifecycle contains the following statuses: New, Draft, Declined, On Review, Stopped, Approved, Active, and Archived.

A promotions dashboard provides information about active and inactive promotions. The most important parts of the dashboard are search in addition to the promotion creation/update feature.

To attract new customers, retailers can look to add gaming or other rewarding elements to encourage an increased level of interaction with the company. For this purpose a Promotion Portal supports different kinds of progress bars, alternative promo conditions like OR, AND or complicated cases with multiple OR and AND conditions.

The marketing team of every retailer is unique and thinks differently. That’s why building an in-house solution is generally more effective in understanding the exact types of promotions and promotion rules that will work best for the brand. In this case, the goal should be for the marketing team to be able to create a promotion and analyze its effectiveness without the need for any external help from the development team or anyone else.

Most 3rd parties can offer a variety of promotions, but there is rarely any dedicated person that can design and implement a tailored promotion or quickly add new rules for a promotion. One more point to remember here is that for the majority of 3rd party providers, you’ll be required to submit a change request every time you want to run a new promotion.

Due to today’s regulatory requirements, the GDPR compliance portal is essential. Every retailer should follow the GDPR, CCPA, and Google rules in order to ensure they are operating legally. The GDPR portal allows users to select a particular customer and clean up all records related to them. All rules and conditions for clean up should be configured beforehand. In order to meet GDPR compliance requirements, every use of customer data must be confirmed by the customer. Therefore every customer needs to understand exactly how their personal data will be used and for what purpose.

The Loyalty program portal allows users to create/edit/stop/archive a loyalty program, view program data, and run multiple loyalty programs in parallel. A loyalty program is a set of rules that determines how customers are awarded points, rewards, and other perks and benefits.

Based on the status or stage of the lifecycle of the loyalty program, it is also possible to update the numbers of tiers, the nature of the qualifiers to achieve every tier, points per every $ spend, rewards conversion rate, pending period, and other types of attributes.

The Loyalty Program Dashboard contains information about every program and the following list of attributes: Program Name, Start and End Date, Tier Name, Points earned per $1, Reward Threshold, Program Limitation, Points Pending Period, and Qualification Rules.

The basic life cycle of a loyalty program contains the following statuses: New, Draft, Declined, On Review, Approved, Active, and Archived. The lifecycle flow of a loyalty program can be configured to meet specific business needs as a part of the program configuration.

Using an in-house program, the marketing team can choose to run multiple loyalty programs in parallel across different regions. Or use different currencies or choose to associate certain aspects of a promotion to a particular region and then assess the outcomes or margins for individual programs to determine if they should be extended or stopped. By contrast, usually 3rd parties offer only one active loyalty program at a time without any ability to link tailored promotions to it.

The CRM is a vital piece of the loyalty picture because the most important part in every loyalty program is the relationship with the customer. At every point in time a retailer should know as much about their customers as possible and this information should be up to date, real time information.

The following basic attributes should be carefully managed for every customer (this list of attributes is not exhaustive, but includes the most important ones):

• Personal information
  • First/Last name
  • Date of birth
  • Shipping Address
• Transaction information
  • All made purchases
  • Returns
  • Pending transactions
• Points activity
  • Earned points per transaction
  • Earned bonus points
  • Earned retailer credit card points
  • Expired points
  • Returned points
  • Manually assigned rewards
  • Total available points
• Rewards activity
  • Earned rewards
  • Manually assigned rewards
  • Birthday rewards
  • Special personal rewards
  • Returned rewards
  • Total rewards
• Tier history
  • Tier level
  • Tier start/end date
  • Description
  • Reason to downgrade/upgrade

• Account status. Every loyalty account should be supported by the lifecycle, so it will be easier to manage current customer status, amount of points, and rewards. For example, the following account lifecycle can be used:
  • Pending – a customer should confirm their phone number or email address to be enrolled in a loyalty program
  • Active – a customer has signed up to a loyalty program and follows all loyalty program rules
  • Inactive – a customer stopped following the rules, didn’t meet the basic requirements, or didn’t make enough transactions in a defined period of time
  • Frozen – a customer was detected as being involved in fraud
  • Closed – the loyalty account was closed by the customer or retailer

Customer support agents should be supported by having access to basic customer information and their activities. At the same time, you should have the ability to manage and assign appropriate roles to the customer agents. The following main features should be provided:

• To add/update/view/remove a customer agent
  • The following attributes should be saved: First/ Last name, Username, Email, Phone number, Role, and Change a Password for an agent function
• To assign a particular role to a customer agent
  • A list of roles should be configured based on the business model
  • A list of functions should be assigned to a particular role
• Search for a particular customer agent

Analytics/Reports are vital in assisting retailers to properly track and manage the results of loyalty programs. This allows you to accordingly change the type of loyalty program or adjust the rules of a selected loyalty program.

Basic reports required to understand the results/margin of the LP and customer engagements are:

• Promotion report – this allows for an understanding of the margin of each promotion, how it was used, and what kind of customers used it
• Rewards report – this allows you to understand what rewards are used most frequently, how customers earned their rewards, and what margin every reward provided
• Tier report – this enables you to track how many customers are at a particular tier and how often customers are downgraded/updated
• Fraud report – it is very important to be able to follow the CSR agents’ activity if they assign any rewards to themselves or add points to friends or family accounts

Marketing and financial reports should be clear, accurate, and transparent. At the same time, they should be easily customizable and easy to update so marketing and finance teams can  add filters to compare for example different quarters, regions, etc. In the case of in-house solutions, there is enhanced flexibility for report generation as you have greater control over the types of reports and tools you can include. Using a 3rd party solution tends to offer only predefined lists of reports and any new reporting needs or changes will again require submitting a change request.

The REST API enables communication with other internal and external systems. To achieve this, the following basic APIs should be exposed:

• Account Activity – API should provide all personal information about a customer: first/last name, date of birth, shipping address, email, phone number, loyalty account number, if a customer has a retailer’s credit card, total earned points, loyalty account status, etc.
• Rewards Activity – API should provide exact information about a customer’s rewards, available rewards, redeemed rewards, expired rewards, special assigned rewards, expiration dates, and a reward description.
• Missing Points for a Purchase – it’s a common occurrence to find that a customer forgets about their loyalty account while shopping. In this case, the Missing Points API should be provided, which allows a customer or CSR agent to add points for a missing purchase.
• Tentative Points – to help further engage customers, a tentative points display can be beneficial. This allows the customer to see how many points, rewards, or perks they would receive for a particular future transaction.

Email/SMS notifications are an important part of the loyalty program. Email/SMS notifications inform customers about rewards, points, the status of missing purchase requests, and any changes to a loyalty account.

The following email/SMS notifications can be generated by the loyalty engine sent directly to customers:

• $ Reward
• Points
• Loyalty account changes like email, phone number updates, etc.
• Missing Points – confirm request
• Missing Points Found – request awarded
• Missing Points Not Found – in store
• Missing Points Not Found – online purchase

File managementis very important as not all of a retailer’s ecosystems can support real time data updates or synchronization between systems. This means that a file management micro service can be used as an intermediate solution. This involves two main processes, each with their own behavior and rules; Automated file processing or Manual file upload.

Each file has its own specific format for processing, for example content or delivery time. For every file a schedule should be set up, for instance – Hourly, Daily, Weekly, Monthly, or Ad hoc.

All of the above is related to functionality and business needs, but at the same time you need to think about non-functional requirements as well. Otherwise during the green week the system might be down. To avoid it, it’s better to be prepared by gathering statistics from previous years and analyzing each and every component to determine peak and max performance.

At this point it’s important to think about general and common challenges like scalability, availability, backups, a disaster recovery plan for critical data, security, compliance, traceability, health monitoring, and A/B testing. As mentioned earlier, business requirements mean that you need to be prepared to store some data for up to a year, and other types for 3 years, or forever. This is why it should be clearly determined how often data backups must occur.

Another key consideration is the preparation of a disaster recovery plan. This process includes considering all aspects of data management including how the loyalty engine is set up to ensure all necessary data is available 24/7. Other considerations should include whether data is stored in a single region or multiple regions and how CSR agent data access occurs across multiple time zones. These factors will all depend on the specifics of the business.

The key building blocks of the loyalty solution we outlined above should help with creating a more modern, sophisticated, and omnichannel loyalty engine for any retailer. It details using open source technologies and taking a microservices approach. Many retailers choose to move away from 3rd party vendor’s solutions because they want to have direct control over customer data, be able to add features and promotions quickly, and change business rules and return policies based on changing market or customer needs.

In our experience, implementation of in-house loyalty engines can be accomplished within several months. The migration of a loyalty program from a 3rd party vendor to an in-house loyalty engine has many benefits and not only improves the customer experience and data governance, but also helps multiple teams including marketing, finance, and merchandising to generate analytical reports built on clear and transparent business logic. It additionally allows for the quick modification of any existing component of the loyalty engine, adding of new features, and a more transparent points calculation and reward allocation process.