- Everyone has tried ChatGPT, and so have we
- We were hoping for AI to turn our huge broker data set into useful content
- Inaccuracies, privacy concerns keep us from rolling out AI-generated content for now...
- ...but we remain excited and are working to learn more about the potental of AI tools
ChatGPT generated a lot of hype in late 2022, but as a business that relies heavily on written content to help our users, we have long been interested in AI-driven content generation. For example, we already tested a couple of text-generating AI tools two years ago. We had some fun in the process, but on the whole we came away unimpressed and continued to make content the old-fashioned way. So when ChatGPT came along, we were super excited and we just knew we had to give it a try. But what exactly were we aiming for?
We at BrokerChooser have collected hundreds of thousands of data points over the years on more than 100 online brokers - things like stock trading fees, how long it takes to open an account, or what features are available in their mobile trading apps. We use these data points plus a smart algorithm to recommend the best possible brokers for you. Some of these data points also appear directly in our broker reviews or in our ‘Best Brokers’ articles.
However, a lot of these data points aren’t visible to our users at all, even though they are valuable information that could be used to provide direct answers to user questions and problems. Writing content around so many data points and so much information is just simply too big a task for a staff of human writers and editors, even when using some scaling techniques. So we were looking to AI to help us scale up content creation even more. In our dreams, AI would look at this big pile of data and churn out thousands of articles or other types of written content for our users to see.
With these big ambitions came some major concerns. Above all, we feared that OpenAI would simply take our data and run with it - meaning that in the process of training it, it would build our data and unique scoring methodology into its model. In that case, the next time someone asked ChatGPT ‘Which broker offers the cheapest options trading in the UK?’, it would provide a correct answer without ever revealing that this information came from us, and without the user ever visiting our site. Obviously not good for our business.
Because of these concerns, we took some precautions before starting to train our own GPT model. We opted out of OpenAI using our content for improving its public model (it’s possible to do this by writing an email to OpenAI customer support). Also, we started out with just a small subset of our data, limiting our experiment to just a few brokers out of the more than 100 we cover.
When training our GPT model (using OpenAI’s API), we proceeded by feeding a lot of prompt-completion pairs into the model. When we were done, we started asking questions and giving it simple instructions for content creation, to test how well it had processed the information we had supplied and how well it could combine that information into useful content.
The results of our test were not quite convincing yet, to say the least. We ran into the same issue as a lot of ordinary users who play around with the public version of ChatGPT: the model often gave us factually wrong answers in perfectly human-sounding, impeccable English. It generated content that we felt could not be published unless it went through a thorough human editor check - making us wonder whether we might as well write all this content ourselves.
We tried a few tricks to improve the results, with little success so far. We pulled some levers and turned some knobs here and there (such as changing the randomness of responses), but saw no significant improvement. We even briefly lifted the opt-out limitations, to no avail.
So what’s next? GPT and similar tools are obviously here to stay, and we remain excited about their potential when it comes to content generation. We need to keep training our model, figure out how to create better prompts, and just keep learning in general about how these AI tools work.
Despite the lack of publishable AI-generated content so far, we’ve had some positive takeaways. The hype around ChatGPT and our own experiments have made the whole BrokerChooser team a lot more conscious about AI, and made AI a part of our everyday thinking and long-term strategizing. We have started drafting and implementing AI policies, including on legal aspects and transparency toward our users. Whatever the next development in AI technology, we’ll now be much better prepared to employ it for our users’ and our own benefit.