Here’s a condensed braindump of everything I know about extracting and using data out of JSON APIs to actually move the needle on your business operational and financial systems.
Yes, lots of buzzwords in there. But every word has been weighted. I don’t really want to waste my time, and neither yours. Here we go.
Brace yourselves, this book doesn’t have many pictures. Buy the hardcover for nice illustrations. (or hire us).
And for the uncompressed version, see the Appendices at the bottom of this page.
Oh yeah, and this post is 100.000% AI free. I, Ivan Pavlov of Pavlov Analytics, a human with no cybertronic implants, wrote and checked every paragraph, word by word, letter by letter. Lots of backspacing (those who know, know). If you would like the ratio of backspacing to total keystrokes or final volume of letters, I will gladly accept a WordPress plugin that monitors this 😉
Enough for the ADO and preamble. Time for the STEAK. MEAT on the BONES.
****************************
So, you have a JSON API. It gives you a JSON response. Meaning it conforms to the JSON (javascript object notation) standard. Not a very complex standard, basically supports just a handful of rules: the curly bracket(s) {}, the square bracket(s) [], the comma, the double quote(s) “, and a few variables types: object, array, string, boolean, number. JSON either starts with a { or a [. If it doesn’t start with one of those, it’s not JSON, or it’s broken. We’ll test for and handle this later.
You need to be able to talk to this API. If you can’t talk to it, you can’t get any of its output. Game over.
So, you need 4 ingredients to be able to talk to your JSON API: the proper syntax for your request, the proper authorization, the proper location, and meaningful data.
The syntax, also called the schema, is what the literal structure of the request & response the API is expecting to receive and provide. You need to know if the API expects you to pass your name in the “name” parameter, or the “fullName” parameter. If you pass “name”:”John Smith” but it’s expecting “yourFullName”:”John Smith” instead, it probably won’t work, or will throw an explicit error, or will give you nonsense – or WORSE, mostly meaningful with a pinch of nonsense – data in return. Good luck debugging that with everything else you have on your plate 😉
Enough of JSON rules – plenty you can Google there.
To turn MASSIVE amounts of JSON requests & responses into business intelligence, you’re gonna need to 1) GET that data somehow, 2) PARSE & convert that data, 3) REPORT – and preferably VISUALISE that data, and 4) come to some kind of conclusion based on that data, which leads to a recommendation that sounds something like “we should do X instead of Y for the next T minutes/days/months until we see KPI K reach value N, because otherwise we’re bleeding $R {your currency of choice} per {hour/day/year}.”
Why is VISUALISE emphasized? Look up the resolution of your eyes’ retina, and then look up how many neurons, connections per neuron, and firing rate your brain has. Then compare that to the number of transistors, connections per transistor, and firing (switching) rate of those transistors of the fastest GPU on the market you can find. Ironic how strongly I stress visualise for a post with no visuals!
To extract data from your API, you need sensible requests, and a way to run and capture responses.
Where do sensible requests come from? THE REAL WORLD. What are you customers asking you for?
If you don’t have those, you’ll have to come up with a way to generate or simulate these requests somehow.
For using REAL WORLD information, you’ll have to log it somehow – every time a customer uses your API, you want to know this. If you can’t capture and store 100% of these requests, set as high a percentage random sampling as your systems will allow.
Great, now you have gigabytes of log files. If you have tera or peta+, stop listening to me as I personally haven’t dealt with those. What do you do with those gigabytes?
You have a couple questions you can try to answer:
- Are my systems internally congruent and consistent with each other? IE do they mesh well, and how much am I leaking and where? For example, if you were expecting to sell 100 lemonades, and thus you prepared 100 lemonades, but then you only sold 90, and 10 lemonades perished, that’s an objective leak of 10 lemonades. DO YOU EVEN KNOW THIS NUMBER??? Start there. Are you sure you’re generating new blood faster than you’re bleeding out?
- Given that you’re not bleeding out critically, what levers can you pull that give you more than they take away from you?
So how does a JSON API help you answer these? Every JSON API request/response is like a pixel on a screen. The more pixels you have = the better you’re able to see. And then your built-in dedicated neural network between your two ears can AT LEAST SEE CLEARLY to start thinking about the answers to these two questions.
PLEASE PLEASE PLEASE make sure you’re not bleeding out first (or being poisoned, or robbed, or sinking – whatever analogy floats your boat LLC). THEN try to “drive value”.
80% of my job is comparing two things with each other.
This basically comes down to only 2 types:
- Current to future
- Current to past
The past we know pretty well – that’s recorded in past files/databases/bank accounts/your vitamin D levels.
The current we have the most flexibility of knowing – as we can ask real-time questions of our systems, which we CANNOT do for our past systems nor the future.
And the future we can only try to predict – but based on our prediction depends our entire well being!
Two common wisdoms come into mind:
- the best (statistically speaking) predictor of weather tomorrow is today’s weather; and
- the human mind is great at predicting small, incremental deviations from the current path but it is horribly inadequate at foreseeing large, sudden jumps.
Doesn’t it make sense to start with at least what we’re good at? Predicting tomorrow’s weather? Or if that’s too hard, the weather 5 minutes from now?
How can we do that if we aren’t measuring the current weather!
So if you’re not CAPTURING CURRENT DATA… start.
Then start LOOKING at it. If you genuinely believe you require anything fancier than Excel and simple scatter charts to start getting value from your data – you’ve been successfully lied to by expensive consultants. My condolences. If you’re in too deep, try asking them how their XYZ calculation method outperforms a simple AVERAGE function in Excel.
OK, back to JSONs. Basically, every time you see a new square bracket – you need a new data object. For example, one company will have many branches. Each branch will have many staff. Each staff will have many tasks. Each task will have many steps. And each of those objects will have their own unique parameters – a company a name & owner, a branch an address & specialty, a staff a name & salary, a task a cost & worth, a step a duration and expense.
Ideally, you want to at least skim your eyes over EVERY SINGLE ONE of those parameters, and combinations and permutations.
Do you have to MEMORIZE them? No! Just spend a few seconds glancing over them! Get your hands dirty! Open a JSON document in Notepad++, pretty-print it, and just scroll line by line asking yourself “what does this line mean, why does it have that value, and what would happen if it went missing?”
“Why do I have to do all this? How does it help me cut costs or raise revenue???”
Remember how 80% of my job is comparing 2 things side by side?
Imagine on the left hand side is how much money you’re making now. And on the right hand side is how much you’d like to make 🙂
Would you rather just have a single number, or every single pixel and every single lever and variable that matters in getting you from HERE&NOW to DESIRED_STATE???
The problem with getting it “roughly right” – is that the big gold nuggets have already been found… and taking the average “roughly right” density of your current bucket of river sand will drown your gold in averages 🙂 you are after the INDIVIDUAL GRAINS of gold…
and for that, you need a metal detector or VERY inventive sieve… get inventing and building!
For every new JSON square bracket – the data inside the curly brackets that immediately follow your square brackets will form a new table in your pseudo data warehouse. Each curly bracket – wrapped object will ultimately have a square bracket-wrapped parent. Once you parse and separate each square bracket’s contents into a new table, you can JOIN them in something like PowerBI or Tableau and start browsing your data & asking questions.
You have to do this for each square bracket on both sides of your comparison – today’s actuals vs yesterday’s forecasts, today’s actuals vs tomorrow’s plan, today’s old-but-reliable production system vs tomorrow’s new-but-uncertain replacement system…
But once you set this up, you can at least start to CONCLUSIVELY ANSWER basic questions like “how much better will the new system be?” and “how well did we predict today’s sales 4 months ago?”
If a vendor quotes you $XXX to renew your support contact for a year – do you KNOW how well your current Excel “temporary” solution covers the gap? It might be better than you think, and certainly better than the Big 4 $$$ “expert” consultants are desperately trying to convince you of!!
At Pavlov Analytics we are all for transparency, certainty, and building night-vision, x-ray measurement & reporting systems that give you OBJECTIVE data so that you USE A RULER rather than a FINGER IN THE WIND to steer your business… and we use high quality steel, not plastic, to make our rulers 🙂
If you’re fed up with GUESSING from not having OBJECTIVE data on your basic business operations – please don’t hesitate to reach out and we’ll gladly share everything we know that could help you, even if we don’t end up working together.
Especially if you’re a Kiwi-owned, Kiwi-operated business in 2025… isn’t our economy fragile and tight enough as is to be steering our ships blindly in the foggy night?
Sincerely,
Ivan
PS. We are still working on the Appendices.