You Need A Process
For twenty years I was responsible for servers that had to stay up. The standard was five nines, which allows about five minutes of downtime a year. You do not get there by fixing problems quickly. You get there by knowing about them before anyone else does.
I have always found it strange that most distributors hold their business processes to a completely different standard.
No IT manager would accept a server whose only alert was a user calling to say it was down. But that is how most distributors find out a critical process has failed. Think about the billing file to the bank, the nightly price update, the export that feeds a customer portal, or the job that pulls vendor acknowledgments into P21. Ask about any of them and you will usually hear the same answer.
It works most of the time.
What "works" usually means
When someone tells me a process works, what they almost always mean is that nobody has complained. That tells you something about the people downstream. It does not tell you much about the process.
In infrastructure, you learn early that things fail in three different ways, and only one of them is easy to deal with.
The easy one is the loud failure. The job errors out, somebody gets an email, and somebody fixes it. Everyone plans for this kind, and it is the cheapest.
Harder is the job that never runs. A SQL Agent job gets disabled during a server patch, and nobody turns it back on. A service account password expires. A scheduled task lived on a server that was retired last year. There is no error because nothing happened, and it is very hard to alert on something that did not happen.
The worst is the job that runs and gets it wrong. The export reports success but sent zero rows, because a join broke when someone added a new ship-to. The price file goes out with last quarter's contract. The job history is green, so everyone reasonably assumes things are fine.
Most homegrown P21 processes can only catch the first kind. The other two turn up later, usually in a different department.
A failure that never threw an error
A client of mine once wanted to make an item more attractive to larger buyers, so they added quantity breaks. Buy more, pay less per unit. There is nothing unusual about that.
The problem was that at one of the breaks, the unit price dropped far enough that the order total actually went down. A customer who bought 10 paid less in total than a customer who bought 9.
Round numbers make it easy to see. At $10.00 each, 9 units is $90.00. If the break at 10 drops the price to $8.50, 10 units is $85.00. The customer walks away with one more unit and pays $5.00 less.
This was not only a margin problem. Revenue went down too, because every order that landed on that break brought in less money than a smaller order would have.
And nothing failed. P21 priced every one of those orders exactly the way it had been set up. No job errored, and no report flagged it. No customer was ever going to call and complain about paying less. It ran that way for many weeks. I only caught it because I was working on something unrelated that involved the same item, and the numbers looked off to me. Once I looked closely at the quantity breaks, the problem was plain. I notified the client.
Fixing that one item took a few minutes. What mattered was making sure it could not happen again, so we put a review in front of every price break before it could go live. The rule behind it is simple. A larger quantity can earn less per unit, but it always has to earn more in total, in both revenue and gross profit, than the quantity just below the break.
Once you write that rule down, you can calculate a floor. For a break at quantity q, the new unit price has to be higher than the old unit price times (q − 1) / q. For a $10.00 item with a break at 10, the floor is $9.00. At exactly $9.00, you are giving away the tenth unit, and anything lower pays the customer to take it. You run the same test on gross profit, using price minus cost.
I tell this story because it did not involve a job at all. It was a setup change that did precisely what someone told it to do. Some of the most expensive quiet failures come from exactly that.
The gap is what costs you
What a quiet failure costs you has less to do with the error than with how long it runs before anyone notices.
If that price break had been caught the day it went live, it would have cost nothing. Caught many weeks later, it had taken a little out of every order that used it. It was the same mistake either way. The only thing that changed was how long it took to find.
So when I hear "it works most of the time," I know it only describes how often the process fails loudly. It says nothing about how long the quiet failures run, and those are the ones that cost real money.
The monitoring system has a name
If you look closely at one of these processes, you will usually find a person doing the job that monitoring should be doing.
She knows the Monday file ought to be bigger than the Friday file. He knows that if one particular customer's orders are missing from the export, the job needs to be rerun after the warehouse closes. None of it is written down anywhere. It is years of pattern recognition, and it is worth a lot.
The trouble is that nobody backs that person up, and nothing alerts them. When they take a week off, the process keeps running, but nobody is watching it anymore. Nothing in the output will tell you that.
Why these processes last for years
If the risk is real, why do these processes last so long? Partly because the costs land in different places. Fixing the process has a price tag this quarter. Leaving it alone has a cost that shows up now and then, under other names, in other departments.
The other part is that the process becomes load-bearing over time. Reports get built on its output. People plan their week around its quirks. Upgrades get pushed back because no one is sure what an upgrade would do to it.
That last part now has a deadline. Epicor's final on-premises release of Prophet 21 is 2028.1, and active on-premises support ends June 30, 2029. Cloud P21 does not allow the direct database writes and server-side scripts that a lot of these processes rely on. The ones that have survived by being left alone will have to change. I wrote more about what breaks in that move here.
What monitoring a business process looks like
I would start by asking a business process the same questions I would ask about a server.
First, did it run? Something should be expecting the job and should raise a flag when it does not happen. You are watching for silence here, not for errors.
Second, did it produce what it usually produces? Compare today's output to a normal day. If Tuesday's invoice file is usually around 400 lines and today's has 12, someone should look at it before it goes out, even though nothing technically failed.
Third, does the output follow the business rules? The format being right is not enough. Every line needs a valid customer reference, pricing needs to match the active contract, and totals need to reconcile to P21. Setup changes need the same scrutiny. A price break should pass the floor test before any order can use it.
Fourth, did the other side accept it? A lot of partners send back an acknowledgment or a rejection. If nobody reads those, the process really ends at "sent," and that is not the same thing as received.
Fifth, who gets told, and what exactly are they told? An alert that just says "job failed" sends someone off digging. An alert that says "14 invoices are missing a PO number" and lists them can be handled in a few minutes by anyone trained on the process.
These are not complicated checks. Put together, they cut the time between a failure and its discovery from weeks down to minutes. They also free your expert from being the monitoring system, so what they know becomes a backup rather than your only protection.
Where to start
Pick the process that would hurt the most if it quietly went wrong for a month. It is usually one that moves money, sets prices, or feeds a customer. Run it through those five questions. If the honest answer to most of them is "someone would probably notice eventually," you have found your first project.
A process that "works most of the time" is usually one nobody is measuring.
If one of your processes came to mind while you read this, request a call. I will tell you plainly what it would take to have it report its own problems, and whether I am the right person to do it.