Scoring Rules and Rule Sets
© 2014 - WWW.DECISIONS.COM
Scoring Rules and Rule Sets A scoring solution takes input and produces a decision - very much like any business rule. The differences between a scoring rule and a normal business rule can be summarized as:
- potential richness of the result (that its not just a result but also can include an explanation of why the result that was given)
- how the scoring decision was conceived (if it was built in a technology that supports it) and constructed
For the purpose of this paper a scoring rule is: A rule that is built using a composition of smaller rules, or other scoring rules, that produces a single output along with the optional explanation of the decisions made that affected the result.
Scoring Introduction All organizations operate because of business rules, policies, and processes. Some of these are simple, but others can be very complex. These complex rules are often a ‘composition’ of a number of individual decisions that are combined or analyzed to produce a much more nuanced and complex decision. When a complex business or scoring decision is made, it is often described in terms like ‘factors’, ‘calculations’ or ‘elements’. These items are combined together to produce a result and an explanation of a result. Lets take a credit score as an example. A credit score is a number. The number has meaning (great credit, bad credit) and has a number of elements that go into it. How much debt? How many late payments? What percentage is the balance against limit on credit cards? How long have you lived in same place? When you get a credit score, you can examine it in detail to find out some of the elements (individual decisions) that make up this score. Scoring Rules are really just more complex and composite rules. Rule/decision engines are tuned to make specific decisions and the process of combining these elements is delegated to the calling application. The application logic that combines up individual decisions or runs sets of decisions/rules is called a ‘rule run harness’. This can be as simple as a piece of logic that runs through a list of rules and executes them to as complex as something that selects rules based on conditions, weighs and combines the results to produce a more complex decision. One of the primary advantages of using a rule engine is that the rules are that the rule definition and execution are described in a way that is equally understandable to business and technical staff members. Business rules can be found, older versions can be examined and the history clear. Rules can be tested and are readable/editable by nontechnologists. This is useful for simple rules (ex. ‘how late can a staff member be before an alert is sent to manager’) but is critical for complex, multi-part decisions like a scoring decision. If a rule has a number of individual decisions and there is information about how a specific decision participates in the overall result - it can get very complex to manage - and having these elements scattered in different technology or buried in programming language makes it difficult for the business stakeholders to have visibility into the actual logic being run versus the defined logic.
Scoring Rule Example
S
coring rule sets are often the key processes in an organization. Decisions that are not as critical to the organization often are not as nuanced or analyzed - therefore can be expressed in an easier fashion. Take the following two decisions as it relates to a university setting: - Is a student going to be offered admission? - Is a library fine due? Both of these are rules, but one of them is a full blown set of rules that make up some score. This score is then compared to allowable values to determine if this student is going to be admitted (or maybe offered a scholarship for really ideal students or offered conditional admission in the case of marginal students). The admission decision may consider a number of factors: academic record, which courses have been taken, academic record in relevant courses, ethnicity and gender, financial status, volunteer work, references, geographic location, standardized test scores, and others. Furthermore, not all of these factors has same level of weight and there are probably subsections of these factors that have minimums that cannot be overcome by high scores in other areas. For instance, even if high scores are made on the SAT test and all other factors are good, there may be a minimum number of high school credits needed for admission. Contrast this complexity with a library fine. This one is simple. If the book is late, a fine is due. The rules around library books are not complex because they don’t need to be and they are not core to the central mission of a university.
Scoring/Rule Set Fundamentals
S
coring rules are a combination of smaller elements that are combined together to give a final result. If well constructed, the elements that make up the scoring decision are able to be understood as to the fact that they influenced the result. For example, your credit score might be affected by elements that you can understand like high balance on credit cards, choppy job history or late payments. Because a score (like a credit score) is made up of individual elements - these elements can be chosen to be published as explanations of the score - or they can remain as a black box. Each of the elements in a scoring rule set have some commonalities: - they all start with the same basic set of data - they all output data used as inputs into the overall result Scoring rule sets can be made up of compositions of other scoring rule sets. For instance, your payment history might be scored and that score result in a score that is an input for your overall credit score. Scoring is fundamentally a concept of the coordination of individual decision points - whether simple or complex - and the processing of these values to form an overall result or decision.
+
-
40
50>10
AVERAGE
Basic Scoring Strategies
POINTS
T
he manner in which the results of individual decision points or sub scores are combined is often as intricate of a process as the individual decision points themselves. Sometimes the interrelation of 2 or more elements provide meaning or one decision point may be a terminal decision in an overall rule set (no, we won’t admit you if you are convicted of a felony - no matter how good your SAT score is) The manner that individual elements are combined evolves in much the same manner as the decision points. The simple and obvious way of combining scores include: - Total Score: Adding up the number of scores. An obvious problem with this is as new elements are added, the total possible score changes providing less meaning to historical results. - Average: The total score divided by the possible score. - Voting: Number of ‘affirmative’ results in the rule set. - Lowest Score: Evaluate for any scores that are below the minimum. A common first step moving past these simple combination patterns is to provide weighting to increase or decrease the importance of any given element in the overall computation. Your work history score may be low, but it only accounts for 5% of the total credit score.
26
42
TOTAL
31
SCORE VOTING
25
The elements of a decision might be better thought of as a tree than a list - meaning that elements might be combined to form up different granularity inputs. If we take our college application scoring example, the score could be made up of 3 primary elements: - Academic Record - Standardized Scoring - Personal Evaluation Each of these might have different weights (personal evaluation making up 40% of total score, for instance) and each might be comprised of a number of different elements or even other scores. An understanding of the overall scoring might involve a drill into a specific branch of the top level groupings to find out the decisions and their weight that influenced it. It was the fact that you dropped out of both boy scouts and off the football team that caused your personal evaluation score to drop. Of course the combining of the elements, logic based inclusions and exclusions, and custom calculations are important. Any rule engine can compute the individual decisions, but processing the results of those decisions and providing filters to figure out which, if any, elements of the decision are to be included as explanation is what separates a rule engine from a scoring solution.
Understanding the Elements of a Scoring/Rule Set T
he elements of a scoring/rule set solution are the same as those of a decision engine. While the rules share some common elements with each other, the way that rules are built or modeled should mirror the way that the logic owners think about the problem. As such there are a variety of different ways that rules can be built out. Below is not a comprehensive list, but does represent a number of different views of how rules can be modeled.
Sentence This is the most basic form of rule. Most rule engines support this, but most of them support it by forcing the business analysts to be programmers by learning a form of structured english. While the method (typing text in that has to be syntactically correct) of creating the rules makes business analysts pseudo-software-developers‌ the resulting structure of the rule is a very valid way to express business logic. These rules can have conditional branches (either/or) as well as a number of conditions. The Decisions Platform rule engine walks an analyst through a series of steps, picking the data target, selecting from a number of appropriate verbs, and finally allowing configuration of any additional elements of the rule.
Table (Truth Table/Rule Table) A truth table differs from more simple rule forms in the fact that there is a set of conditions (think data target/subject and verb) that are configured in columns. Then there are a number of rows that represent the matching criteria and the result for this criteria if it is met. Table based rules have the added benefit of being able to have more than one resulting answer if desired. These rules are often used to look up appropriate data or rates, to find actions that need to be taken or produce a set of further criteria that might be useful in determining the further processing in the application.
Diagram A rule diagram that lays out the same elements that would be in a sentence style rule, but shows it in a graphical view. In the decisions rule engine a rule author can toggle back and forth between a diagram and sentence view.
%
Intersection The intersection builder allows building of 2 different axis of rules that meet at a point - which has ability to configure an outcome value. An example would be determining commissions by deal size and years at company of the sales person. Two different rules that intersect at a point.
Procedural/Flow Based There are some rules that cannot be expressed in a declarative fashion, but have multiples stages of processing. This is where using the ‘process engine’ to process data as a rule allows for much greater flexibility. It is critical in the context of a rule engine that there is a pattern that lets you model the 5% of rules that are just too complex without forcing a paradigm change. A flow, running without user interaction shares the same basic pattern as a rule does. Data comes in, a result is returned. In the Decisions platform, flows can be made to run ‘as rules’, and flows can be embedded into a rule as an additional verb.
Expression/Calculation Formalizing any mathematical calculations also is a representation of business logic. Certainly things like rates may sometimes be applied using declarative rules (if loan amount is below 5,000, the rate is 2.9%) but there are times that an ‘excel style’ math computation is useful as well. These computations can be used like any other rule to provide data to calling application or process.
Score Rule Sets Scoring elements can be scoring rule sets themselves. This is a very common pattern that allows similar granularity of decisions to be combined and managed as logical units.
Injecting of Data into Scoring/Rule Executions W
hile by definition there is a basic set of data that is presented to the scoring system overall, the elements of scoring may need to gather additional information or incorporate external systems logic to provide be able to make the decisions needed to participate in the rule set. Because the Decisions Platform is built on a workflow/integration engine, these integrations are also done graphically using the same pattern as rule and score constructions. If data is needed from an external system as part of a rules execution, it can be fetched and cached when the rule needs it.
Parameterized Rule Sets
?
O
ften scoring is used to make more than one decision or can be applied in more than one context. To use the university admission scoring example again, there may be different sets rules for the college of engineering and the college of education. They may share some common elements (basic criteria) but might have different thresholds or inputs into those rules based on the context that its being asked. A high score on the math part of the SAT’s is going to produce a different result in engineering or education. There are two main scenarios that inter-relating rule sets is practical: - Where there is shared rules between two sets of logic but there can be adjustment between each of them. - Running scores against a number of different scenarios to find the best match (highest score?) is desirable. For instance, I could score an insurance application against a number of different policies and determine which one is the best fit. Adjustments may include excluding a rule or providing a different comparison value to a rule to determine what is needed. Additional rules can be added at the terminal nodes of the parameterized set as well.
After the Score If a scoring ruleset is to be useful, it needs to be run from some active context, like a website or an application. Users will rarely interact directly with a scoring rule engine because a rule engine lacks the interfaces and the context of what to do with the resulting evaluations. Applications can use a service interface to call the scoring rule set. The result of a scoring decision can be handled with a workflow - or workflows can be used to gather additional information by user or system interactions for the purpose of re-running scoring decisions. Workflows process information through a series of steps with people interacting at some points and software automating at others. Rules can be used as conditions to inform on the paths a workflow takes. Consider the example of processing medical lab results in a hospital. This “workflow” might have the following high level steps:
1. Read result data from a database in the lab. 2. Score the patient (diagnosis, history, environmental factors, age, gender, ethnicity) for conditions that might indicate a medical emergency. a. Based on score, alert medical staff. 3. Evaluate the data for possible additional tests that could improve patient care. 4. Get the Doctor’s review and approval of the results. 5. Store the lab results in the patient’s electronic medical record. The above scenario is a simple example of a workflow that uses a scoring ruleset. The workflow both initiates the scoring after gather the data and handles the results (routing, followup, storage) of the process.
Why not just use programming languages to do scoring? Business rule engines produce logic that can be combined into scoring solutions. Programming languages also produce logic‌ so, why not use a programming language to encode your scoring logic? There is no rule engine that can produce logic in a more flexible way than a programming language - however, there are really good reasons to consider having the core rules of the system constructed using a rule engine. When rules are built in a rule engine, they become a ‘formal artifact’ that is able to be named, classified, evolved and searched. When rules are separated out of the structure of the system, they can be understood, discussed and evolved. Building logic in programming languages also requires programmers to build the logic. This implies that the programmers have to understand the details of the rules to the same level as the business experts and owners who craft the rules. Entire disciplines of technology are dedicated to solving the problem of business people and technical people understanding each other - and often this has limited success. Since software is often built in layers, and these layers are implemented using different technologies, its very common to have rules in more than one spot in software stack. The same rule that is built into site/ui to provide user feedback, could also be in a stored procedure to run on a database and in the middle tier programming language. The real reason that business rule engines are often ideal is because it allows business people to build, control and understand the rules. Rules that are embedded into structural logic of data movement, in complex formulas or stored procedures in databases are hard to find, understand and comment on by the people who are responsible for designing the rules.
Characteristics of Scoring/Rule Technology O
ne of the ironic things about many business rule technologies is that they are just another form of a programming language - however, a less powerful and more constraining one than software developers use. Business analysts would need to learn a particular script or syntax to be able to create or modify business rules. Business rule technologies have many benefits, however, in order to realize these benefits, the rule engine should have a few basic characteristics.
No Code/No Script No Structured Language The key aspect of a rule engine is that it must allow construction of rules by non-programmers. There may be different interpretations of what this involves, but having an environment where a business person can assemble elements into a rule without writing code is a requirement. A rule designer that gives a text area for a person to type in statements - is really a programming environment, not a business user focused tool.
Rule Integrity Checks A rule must be structurally sound in order to operate. At a very basic level is this means a rule has all of the pieces that are required to make the rule operate. Operations must match the datatypes that they represent. All outcomes must be present, etc. This integrity check must be able to tell a rule writer where the problem is and guide them to a solution in which to resolve it.
Variety of Representation Models This is covered in more detail below, but there are multiple ways to think about expressing a rule and a good rule engine will allow the construction of a rule in a manner that matches how a rule is understood rather than forcing it into a more constraining model.
Integrated Testing Obviously knowing a rule is technically valid is a key element, but how do you ensure that the rule produces the results that are desired without having to drop into a more technical tools to validate. The tester must have ability to run rule with different values as inputs and evaluate the outputs as well as see the execution path and data.
Catalog/Rule Management One of prime advantages of using a rule engine is the rules are clearly spelled out and understandable - but they also must be able to be searched, classified and viewed in a consistent manner.
Versioning/History It is not only important to know the current state of the rule, but also see versions as the rule has changed over time. Being able to view and also run against rules as of a point of time is important to understand the changing results in the business logic overtime.
Unit Testing
Composite Rule Structures
Testing a rule should be able to both be done in an interactive manner and also have test cases that can be built and run automatically to ensure the rule is behaving as expected.
Scoring requires combining rules into sets or hierarchies that can be run as a single unit. These structures should enforce the same rule inputs and outputs as well as give metadata about the lifecycle (dates, etc), participation (weight, order) and description within the overall set and context. A rule engine without the combining of rules into rulesets may be useful for processing individual decision points but falls short of allowing complex, composite decisions such as scores to be calculated.
Extensibility New rules elements need to be able to be added to do things that are specific to business problem. While many rules can be constructed out of standard pieces sometimes a rule element will require access to some information, service or functionality that is not present in the ruleset. An SDK for developers to create new pieces is needed - and these pieces should operate and be integrated like the pieces that come native with the rule engine.
API Access Rules need to be run by something to be useful and used. Processes, websites, applications need to be able to get access to the rules.
*Bonus* Workflow Engine Integration Ok, this is a bonus‌ but if the policy/rule is built graphically, having the option to automate the workflow around it graphically to process data, involve users, do assignments, etc. should also be considered. The Decisions Platform combines a graphical rule builder with a flow builder using the same principles and commitment to graphical logic creation.
Checklist - Do I need a Business Rule Engine? Obviously, Decisions is a big proponent of technologies that allow businesses to more quickly evolve their operations, rules, etc. While our technologies are focused on empowering business analysts, we understand that no single approach is best for all situations. Here are some simple questions to ask in determining if a business rule technology is a good fit for your situation:
Is there an advantage of your rules being able to be created/edited and tested by business analysts rather than programmers? Is there a need to understand the rules that apply to a specific interaction? (ie, this claim was denied because of rule a, b and c) Do you need to know what rules are applied to a certain transaction at a specific point of time? Do you want to be able to test rules outside of the context of the application to ensure the logic is right? Is it important that non-programmers can understand what the actual logic is - not what the logic was supposed to be? Do your rules change on a consistent basis, or do they need to change rapidly? Do you have checklists just like this one that are used to evaluate things in your organization? Are there manual or automated processes and workflows in your organization that need ‘thresholds’ applied to them?
© 2014