Automation Monitoring Dashboard: What to Track
An automation monitoring dashboard is one screen that shows whether your automated workflows are running, running correctly and running fast enough to protect the business. For an order desk, that means four things in one view: how much work is coming in, how much is posted to your ERP, how much is stuck waiting for a person, and how much is wrong. Build it around those questions, not around technical logs, and everyone from the CEO to the sales engineer can read it in seconds.
We built this guide for wholesalers, manufacturers, purchasing managers and technical sales teams who receive orders, inquiries, invoices and PO confirmations by email and want to see what automation is doing with them. If you use Ampwise, the same logic applies to the status tracking and approval steps you already work with.
TL;DR
- A useful automation dashboard answers “is it healthy right now, is it accurate, and what needs a human?” in about five seconds.
- Start with four live numbers: volume in, success rate, queue depth and age of the oldest waiting item.
- Track accuracy separately from speed: validation errors, manual corrections and exceptions tell you more than a green “completed” count.
- Alert on queues building up, not only on failures, because slow problems never trip an error alarm.
- Build one layer per audience: a live operational view for the order desk, a short KPI summary for executives.
Table of Contents
- Monitoring versus observability: why the difference matters
- The four numbers that belong at the top
- Accuracy metrics: the part most dashboards skip
- Speed metrics that predict customer experience
- Monitoring purchase-order confirmations
- Alerts: watch the queue, not just the errors
- Designing for each audience
- Does the five-second rule hold up?
- A simple build sequence
- Perspective
- See your email-to-ERP workflow in one place
- FAQ
- Sources
Monitoring versus observability: why the difference matters
Monitoring tells you what happened: a job ran, a job failed. Observability connects that execution data to what it means for the business: which orders are late, which customers are affected, which service level you are about to miss.
The distinction is practical. An Enterprise Management Associates paper, Smarter Automation Through Observability, notes that most organizations run several automation platforms and still lack unified visibility into how they interact, when service levels will be breached or which business services are at risk. It also says traditional monitoring reports what happened but does not predict what is coming. The paper describes teams that are “reacting, not orchestrating”: a failed dependency delays an overnight report, nobody notices, and root-cause analysis takes days instead of minutes.
In an email-to-ERP setting the same blind spot looks like this. A customer sends a purchase order as a scanned PDF on Monday afternoon. The extraction runs, one line item fails validation, and the order sits in a queue. Nothing has crashed, so no alarm sounds. On Wednesday the customer calls to ask where the order is. A dashboard built for observability would have shown that order aging in the queue on Monday evening.
So when you design your dashboard, treat “automation observability” as a simple test: can I see, without digging, which business items are affected and how long they have been waiting?
The four numbers that belong at the top
The most useful layout we know starts with a small row of headline numbers. A Hookdeck guide to production observability recommends a top row of total request rate, overall success rate, current queue depth and the age of the oldest pending event, with 24-hour charts underneath. It was written for webhook systems, but the four numbers transfer directly to order processing:
| Dashboard number | Order-desk meaning | What it tells you |
|---|---|---|
| Volume in | Emails with orders, inquiries or invoices received today | Whether today is normal, busy or unusually quiet |
| Success rate | Share of documents extracted, validated and ready to post | Whether the automation is reading your documents correctly |
| Queue depth | Items waiting for review, approval or a fix | Whether work is piling up |
| Oldest item age | How long the longest-waiting item has sat | Whether any customer is being left behind |
The 24-hour trend chart matters as much as the number. A queue that jumps from 0 to 500 is probably a problem. A queue that moves between 400 and 600 all day may be normal for your business. Without the trend line, the same number can mean opposite things, so always show both.
Accuracy metrics: the part most dashboards skip
A count of “completed runs” flatters any automation. An order can be completed and still be wrong. If your goal is fewer data entry errors, accuracy metrics deserve as much space as throughput.
For order and invoice workflows we suggest these:
- Validation error rate. The share of documents where a required field, price, unit of measure or customer reference fails a check before posting.
- Manual correction rate. How often a reviewer changes extracted data before approving it. A rising rate usually points to a new document layout or a new customer format.
- Exception rate by cause. Missing item numbers, unknown customers, price mismatches, quantity mismatches. Grouping by cause shows what to fix.
- Duplicate rate. The same order arriving twice, for example as an email and a forwarded copy.
- Rework after posting. Orders reopened or corrected in the ERP after release. This is the number that exposes errors the review step missed.
Practitioner guides on automation measurement agree on the principle: set a baseline before you automate, then compare accuracy and turnaround against it. You cannot claim an improvement you never measured. Take two weeks of manual-process data, such as minutes per order and error rate, and put it beside the automated figures on the same page.
If you want the deeper logic for validating a purchase order before it reaches the ERP, our guide to validating purchase orders before ERP release walks through three checks you can turn into dashboard indicators.
Speed metrics that predict customer experience
Speed on a dashboard has two faces. Handling time is how long the work takes once someone touches it. Elapsed time is how long the customer waits from the moment the email arrives. Automation usually shrinks handling time dramatically, but elapsed time is what the customer feels, and it can stay long if items wait in a queue for review.
Show both:
- Time from email received to ERP-ready. Your core cycle time.
- Time from ERP-ready to approved and posted. The human review segment.
- Time to first response for inquiries. The metric technical sales teams care about most, since quote speed often decides who wins the job.
- Percentage processed within your target window. For example, orders posted the same business day.
If you need a fuller set of operations indicators, including dwell time, see our piece on order processing KPIs that cut dwell time.
Monitoring purchase-order confirmations
Purchasing managers have a specific job for the dashboard: knowing which POs the supplier has acknowledged and which are still open. Splunk’s guide to monitoring purchase order life cycles lays out a model built on EDI transaction types: the PO itself (EDI 850), the functional acknowledgment (997), the order confirmation (855) and order changes (860). It recommends KPIs for acknowledgment timeliness, supplier response time, order error rates and unacknowledged open POs, plus a real-time alert if an acknowledgment has not arrived within 24 hours of transmission.
You do not need EDI to use the idea. Whether confirmations come as EDI messages, PDFs or free-text emails, the dashboard questions are the same:
- How many POs went out today, and how many have a confirmation?
- Which confirmations are older than your threshold, say 24 hours?
- Which confirmations differ from the PO in price, quantity or delivery date?
- How long does each supplier typically take to respond?
The third question is where email-based workflows hide risk. A confirmation that arrives with a changed delivery date is easy to miss in an inbox. If your system extracts confirmation data and compares it with the original PO, a mismatch counter on the dashboard turns a buried email into a visible task.
Alerts: watch the queue, not just the errors
Most teams start with failure alerts: notify me if the error rate passes 5%. That catches fast, loud failures. It misses the slow ones.
The Hookdeck guide makes the point well. A destination that responds correctly but slowly will grow the queue without raising a single error, and by the time timeouts push the failure rate up, you are already hours behind. It suggests alerting on queue depth, for example when pending events exceed 1,000 for more than 5 minutes, alongside error-rate alerts.
Translated to an order desk, a sensible alert set looks like this:
| Alert | Example trigger | Why it matters |
|---|---|---|
| Queue buildup | More than a set number of items waiting for review for over an hour | Catches slow degradation before customers notice |
| Aging item | Any order unreviewed past your service target | Protects individual customers |
| Validation spike | Error rate well above its normal range for the day | Often signals a new document format |
| Missing confirmation | No supplier confirmation within 24 hours of the PO | Prevents surprise delays |
| Silence | No incoming orders for an unusual stretch of a normal business day | Can mean a broken connection or mailbox issue |
Tune thresholds to your own volume. A wholesaler with 40 orders a day and one with 4,000 need different numbers, and an alert that fires constantly gets ignored. Review each alert after a month: if nobody acted on it, raise the threshold or retire it.
Designing for each audience
One dashboard rarely serves everyone. The better pattern is layers, each built for a person and a decision.
The order desk and operations team
This view is live and granular. It shows the queue, the oldest items, exceptions by cause and anything blocked. It should lead with the current state rather than last month’s trend, and use large status indicators with minimal clutter, as Domo’s dashboard design guide recommends for operational views. The reader’s question is “what do I work on next?”
Technical sales
Sales engineers need to know whether an inquiry has been answered, how long it has waited and which customer it came from. A short list sorted by age, with the status of each request, serves them better than charts. Our article on inquiry response automation covers how email-heavy teams speed up replies.
Purchasing managers
The PO confirmation view described above: open POs, unconfirmed POs, mismatches and supplier response time.
Executives
The CEO view is a short summary: orders processed, share handled without manual entry, average turnaround, error and rework trend and cost per order. The same Domo guidance suggests executive dashboards show roughly three to seven KPIs above the fold, so scanning stays fast. Executive views favor the “so what” over the detail. If your sponsors ask about return on investment, pair this view with the baseline you took before automating, and see our overview of ERP automation ROI for how to frame payback.
Does the five-second rule hold up?
The five-second rule says a viewer should grasp the key message of a dashboard within five seconds. It is a guideline, not a law, and Domo’s executive dashboard guide applies it to executive dashboards in particular. In practice it works as a design test. Show your dashboard to someone for five seconds, hide it, and ask: is the workflow healthy, and what needs attention? If they cannot answer, you have too many tiles, too many colors or the wrong number at the top.
For operational views, where staff stay on the screen all day, the rule matters less than clarity of grouping. Even there, put the “is anything wrong?” signal in the top-left corner and keep detail one click down.
A simple build sequence
You can set up a useful dashboard in a week by working in order.
- Write down the decisions. List what each person needs to decide with the dashboard: what to review next, which supplier to chase, whether to add capacity.
- Record the baseline. Capture minutes per order, error rate and turnaround before automation, or from the last period of manual work.
- Choose five to eight metrics per audience. Include at least two accuracy metrics and at least one metric about waiting time.
- Define each metric in plain terms. What counts as “complete”? What counts as an “exception”? Put definitions in a tooltip or a short note.
- Set alerts on queues and ages first. Add error-rate alerts second.
- Segment. Break results down by customer, supplier, document type (email text, PDF, scan, Excel) and order type. An average hides the one customer whose format keeps failing.
- Review weekly. Look at exceptions by cause, fix the top one and watch the rate fall.
If you are choosing the operational indicators that fit a status-driven business, a practical example from another industry is tracking job status, where visible status stages replace chasing people for updates. The same idea applies to orders: every item should be in a named stage, and the dashboard should count items per stage.
Perspective
Dashboards fail when they are an afterthought bolted onto a process nobody can see into. Our view is that visibility should come with the workflow itself.
That is how we think about Ampwise. Orders, inquiries, invoices, PO confirmations and incoming goods arrive in Outlook or Gmail as emails, PDFs, Excel files and scans. We extract and validate the data without templates, label the emails, track their status and extract metadata, and let your team approve verified data into the ERP with one click. Because every item has a status, you can count items per stage, see what is waiting and see what has been posted.
That is also why we recommend keeping a human approval step in the loop and measuring it. The approval queue is your most honest accuracy signal: when reviewers stop correcting, you know the automation fits your documents.
Our aim is a workflow where the dashboard questions, such as what is waiting, what is wrong and what is late, are easy to answer because the work is already organized that way.
Evert
See your email-to-ERP workflow in one place
If order processing starts in a shared inbox, the simplest way to get an accurate dashboard is to automate the step where the data enters your systems. Ampwise reads orders, inquiries, invoices and PO confirmations from Outlook or Gmail, handles PDFs, Excel files, scans and free-text emails without templates, and sends verified data to your ERP after one-click approval. Smart labeling and status tracking give your team a clear view of what has been processed and what is waiting. You can read more on the Ampwise page or learn how email-to-ERP automation works.
For a closer look at the human-review step that feeds your accuracy metrics, read about automating email orders with review before posting.
FAQ
What is the difference between automation monitoring and automation observability?
Monitoring reports whether automations ran and whether they failed. Observability links that execution data to business impact: which orders are delayed, which service levels are at risk and why. For an order desk, the practical test is whether you can see the age and status of every waiting item without searching logs.
What metrics should an automation monitoring dashboard show?
Start with volume, success rate, queue depth and oldest-item age. Add accuracy measures such as validation errors, manual corrections, duplicates and rework after posting, and speed measures such as time from email received to posted. Keep five to eight metrics per audience so the view stays readable.
How many alerts should I set up?
Fewer than you think. Begin with queue buildup, aging items, validation spikes and missing confirmations. If an alert fires and nobody acts on it for a month, adjust the threshold or remove it so the important ones keep their weight.
What is the five-second rule for dashboards?
It is a design test: a viewer should understand the main message of the dashboard within about five seconds. Use it to check that your top row answers whether things are healthy and what needs attention. It suits executive views best, while operational views add detail one click below.
Sources
- Smarter Automation Through Observability
- Webhook Observability Architecture for Production Systems
- Monitoring purchase order life cycles
- Dashboard Design Best Practices: Layouts & Examples
- Executive Dashboard Best Practices: KPIs and Design
- How to Track Job Status in a Garage Without Chasing Techs
