For a team of AI fanatics like us, it’s been a thrill to see the AI market take off over the last 12 months. The barrier to entry is getting lower for AI builders. The prohibitive expense of training new models is an upfront cost that is increasingly being fronted by closed source API providers like OpenAI, as well as researchers and projects building open source foundational models such as Stable Diffusion, Whisper, LLaMA and others.
Even with upfront training expenses drastically reduced, we continue to hear that the long term compute costs of production deployment threatens the economic viability of any AI offering. And that’s if the developer or enterprise can even get access to the AI compute they want to create their app/service in the first place.
At OctoML, we are on a mission to deliver affordable AI compute services for those who want control over the business they are building. That’s why we built a new compute service, available now in early access. It delivers AI infrastructure and advanced machine learning optimization techniques that you can only find in large scale AI services like OpenAI, but gives you the power to control your own API, choose your own models and work within your AI budget.
Early access users can try the fastest Stable Diffusion 2.1 model (with no change to the accuracy/performance of the model) on the market, without needing to train or retrain the model. Here is some early data that demonstrates the performance gains:
Stable Diffusion runs blazing fast on A10Gs why are you waiting on A100s?
We are hearing time and time again from AI developers that GPU availability is hampering their ability to create their new AI-powered app. When we double-click on these conversations, we are finding that organizations are taking it on faith that only newer NVIDIA hardware i.e. A100s deliver the price/performance they need to run their models at scale. That’s why we are excited to share that A10Gs can deliver the right user experience–1.35 seconds to generate an image–that any mainstream Stable Diffusion powered-app needs. And most importantly A10Gs ARE available everywhere and aren’t being rationed.
Not only is OctoML’s optimized version of Stable Diffusion 2.1 blazing fast, it actually outperforms by 30% the best in class do-it-yourself configuration available to sophisticated users (those who have experience in machine learning engineering). The DIY configuration running on the A100 uses the xFormers package from Meta that leverages leading edge memory efficient attention implementations, fused kernels, and other sophisticated techniques to get high performance on GPUs running on the beefiest hardware (assuming you can get it). Even with that level of DIY sophistication and running on A100s the OctoML version of Stable Diffusion 2.1 actually outperforms it on less powerful hardware.
How does OctoML stack up against hosted services? 3x faster for 5x less money.
When running AI in production, hosted services like HuggingFace (Inference Endpoints) are popular options because they’re easy to use and reduce the headaches of manual deployment and infrastructure management. Now that we’ve grounded you in the fact that you don’t need the latest/greatest NVIDIA hardware to run your models, let’s compare to HuggingFace, which is the most popular distribution source for Stable Diffusion.
Whereas the HuggingFace version running on their Inference Endpoints–that infrastructure has been designed for and optimized for the ML researcher community–has not been developed to deliver best-in-class compute services.
As a proof point of that we highlight that our Stable Diffusion model hosted in our compute service has a speedup range between 2X on a lower end image quality (512x512, 30 steps) to 3X better at the very high image quality (768x768, 150 steps).
2.6X speedup on lower quality images
512x512, 30 steps
3.6X Speedups in high quality images
768x768, 150 steps
Battle of the AI apps
OctoML gives you the choice and flexibility to tailor price and performance for your use case and the experience you want to deliver your user.
It can also give your business big competitive advantages.
Imagine you have an application running Stable Diffusion from OctoML, and a competing AI-powered app is using Hugging Face. It would take them nearly 4X longer to deliver a customer an image at the same quality– nearly a minute! And it would have cost them 5X as much as you would be paying using OctoML. If your image quality requirements are flexible, you could deliver images with OctoML for under 1/100th of a cent.
Another unique aspect of the OctoML approach is that unlike other solutions on the market, it supports friction-free fine-tuning to customize Stable Diffusion against your own data sets. Other approaches require that the model be recompiled every time there is fine-tuning which in the case of doing so with TensorRT can take approximately 30 minutes.
To get early access to the OctoML compute service early access, please sign up here.
If you are interested in also working with us on your fine-tuning requirements/needs, or if you have a use case to leverage our Stable Diffusion model outside of our compute service because you built your own serving infrastructure, please contact us here so we can schedule a time to talk.
Related Posts
Generative AI is the hottest workload on the planet, but it’s also the most compute intensive, and therefore expensive to run. This puts startups building generative AI businesses in a tricky position. Not only must they deliver killer product experiences that grab attention and market share – they need to make the economics work too. To lower compute costs, generative AI models need to run faster and more efficiently on a more diverse set of hardware.
It is an objective fact that there are not enough super powerful GPUs for all who want them. But is that all the compute there is? Of course not. The shortage we’re witnessing is, in fact, a highly inefficient use of GPU compute, which cuts against the grain of how clouds operate and are consumed.
