How to evaluate AI Solutions As A Business?

(And not as an Investor)

AI as a technological revolution is probably going to be at the scale of the internet, let’s start off by getting that out of the way, but as the veterans would remember the Internet revolution also had its fair share of problems.

Disclaimer: This is not relevant for businesses whose core function is AI, for them ( and us ) AI is very much an investment and should be embraced as such.

This is for the companies who want to adopt AI or are skeptical to adopt AI.
The problem I see today draws a pretty good parallel to the dot-com bubble. In the current scenario most companies can be divided into having two opinions regarding AI.

  1. AI is a silver bullet for all problems.
  2. AI is not yet mature enough for meaningful adoption.

And as most things the truth lies somewhere in between. AI is definitely not a silver bullet (which people are realising as the Gen AI hype cycle moves towards the plateau) but it is nothing to sleep on as well.

So the question is how do you cut through the noise and adopt AI in an effecting way?

Here are some pointers we look for when designing an effective AI solution:

What makes an Effective AI Solution?

1. The problem should come first (and I mean this chronologically)

The problem you want to solve, should exist before the plan to solve it. Simple enough, but as human beings, it is hard to resist the temptation of something novel and the fear of missing out on something great.
I believe the painkiller and supplement analogy helps with this.

If you see an exciting AI project and think that would be good to have in our organisation, then you are buying supplements, when you should be focusing on getting painkillers.

2. Solution should be worth it, in the long term

When evaluating the solution, the recurring running cost (architecture + support + maintenance) and only the running cost should be compared to the cost of the problem persisting.
If the saving is significant enough it is a likely a good solution.

Note: I would try to avoid factoring LLM costs too seriously since they are most likely to keep getting cheaper.

3. Measurable Impact

If your organisation is skeptic of AI, telling them it might pay off, at some point in the future is not going to help. It might be true ( and most likely is ) but it isn’t comforting.
Metrics are your friend. Metrics should be defined before hand to judge the success of the POC and the solution

What do you need to build an effective solution?

Data, Data, Data

With generative AI making ML/AI solutions even more accessible than before, the only differentiator you have is your data.

Storing and maintaining the Data, responsibly

Data governance and engineering are key components in achieving Responsible and Effective AI solutions.
Make sure to invest in your data pipeline and engineering. It will give you returns in both the short term and long term.

Internal upskilling

Even if you intend to outsource the initial development and support it is always a good idea to have a small internal team that can guide you through the process and help you utilise the solution internally

Stakeholder buy in.

This is crucial and is surprisingly bottom up and not top down. You want the people who will use the solution to actually get value from the solution so you need them onboard when designing it.

Next Steps

Invest in Data

Even if you are skeptical about the current usefulness of AI, it will not hurt to implement a robust data engineering and governance framework in your organisation. In the long run, it will allow you to be ready, with an advantage of proprietary data that no one else has, when AI hits a maturity of your liking and in the short term it will allow you to better extract insights across your organisation.

Identify potential use cases

You know your problems best, and if you can identify what your bottlenecks are and where you could potentially use AI in the future you will be better ready to filter the noise around AI and won’t fall prey to the hype cycle.

Solve then optimise

When planning to implement a solution don’t go all in and try to develop your own little AGI, instead first solve the problem for a fraction of your targeted users with the least effort possible and then scale and optimise for the organisation.

For example, instead of jumping in to finetuning a Large Language Model like GPT (or heaven forbid training your own) from scratch using data you don’t have, build a prompt engineered, RAG LLM agent that can solve it to a reasonable amount. If this works you can then look into optimising it with finetuning as well.

Bonus Tip: If you collect feedback on the AI agent from your users, you can generate the data required to optimise the solution, while using the solution.

Conclusion

Adopting AI in your business is not just about staying ahead of the curve; it’s about solving real problems in a way that is sustainable and impactful. The hype around AI can be overwhelming, but by focusing on your specific business needs, evaluating long-term value, and building a robust data foundation, you can cut through the noise and make informed decisions.

Remember, AI is a tool—a powerful one—but like any tool, its effectiveness depends on how you use it. Start small, measure impact, and iterate. With the right approach, AI can become an integral part of your business strategy, driving efficiency, innovation, and growth.

Leave a Reply

Your email address will not be published. Required fields are marked *