Gemini 1.5 Flash, GPT-4o, and Other LLM Prices

Google and OpenAI have made a ton of announcements related to new AI over the past two days.

Generally, there are two main aspects that people are curious about when it comes to using them.

Performance and price. (Of course, there are many more features, but I'll leave those to the expert bloggers to review...)

Open AI - GPT

As always with OpenAI, the new GPT-4o is cheaper than the previous GPT-4. There are tons of reviews out there if you want to look into the performance, so I'll just focus on the pricing here.

Basically, OpenAI has been lowering the price every time they release a new product since the initial launch of GPT-4 after GPT-3.5 Turbo. Naturally, the performance has also been upgraded. Currently, if you're looking for the most affordable option, GPT-3.5 Turbo is the way to go. For all other scenarios, it looks like GPT-4o is the best choice.

Anthropic - Claude 3

Anthropic hasn't released any new products recently, but they're still a significant LLM company because of Haiku, which offers great value for money, and Opus, a high-performance model.

Based on input token pricing, Haiku is the cheapest of the three, making it the most affordable option for simple text processing tasks.

In fact, until Gemini Flash came out, Haiku actually outperformed Gemini 1.0 Pro, making it a highly useful LLM.

Google - Gemini

Google maintains two pricing systems.

One is AI Studio, and the other is Vertex AI.

AI Studio, like other companies, uses token-based pricing, while Vertex AI uses a unique character-based pricing system.

Based on the table above, if 1 token averages less than 3 characters (1-2 characters), using Vertex AI is cheaper. If it's 3 characters or more, AI Studio is cheaper. But usually, English text will definitely exceed that character count, making AI Studio the more economical choice. These days, Korean text often has multiple characters per token as well...

Anyway, looking at input tokens and performance alone, Gemini 1.5 Flash is significantly better than 1.0 Pro, and for high-performance tasks, 1.5 Pro is the better option.

Summary

In terms of performance alone, based on MMLU, it seems like GPT-4o > Opus > 1.5 Pro.

For highly intellectual tasks, GPT-4o is the best. If you want something a little cheaper or need more powerful features than GPT-4o, Gemini 1.5 Pro might be a good option. The actual experience can differ slightly, so you should choose the one that suits you best.

If you need to process a large amount of text at a low cost, you have two choices:

If the ratio of input to output is low (e.g., when you need to input a large document and output a short result), Claude 3 Haiku is the cheapest. However, Haiku has high output costs, so conversely, if the ratio of output is higher (e.g., when you input a specific text and then instruct it to modify or change it), I recommend Gemini 1.5 Flash. In such cases, Flash has the lowest output cost.

Summary and Conclusion

"I don't care about the price, I just need the best for the most complex tasks." -> GPT - 4o

"But, my input token size exceeds 128K." (GPT - 4o only supports up to 128K) -> Opus

"I need high performance but at a lower price, or my token size exceeds 200K." (Opus only supports up to 200K) -> Gemini 1.5 Pro

I need the cheapest LLM possible. -> Haiku

But, the Input/Output ratio has a slightly higher output or exceeds 200K tokens. -> Gemini 1.5 Flash