With the advances in artificial intelligence and machine learning, search engines are becoming more sophisticated and can provide more relevant and personalized results to users. Despite increasing criticism, Google still dominates the search market. However, it's worth taking a look at where we might be heading.
The useful information on the Web is now concentrated in a few places like Reddit, Stack Exchange, Wikipedia, etc.
Most people still use Google as the starting point for their search. Part of the reason why is that other platforms have pretty bad proprietary searches. Surfacing relevant content that answers a search query is a complex task and not the main focus of most platforms. That's one of the reasons why many people add "Reddit" to their search queries, even if they could just search on Reddit itself.
What enthusiasts and authentic users say is far more valuable than an article that was made for views by some corporates.
Redditors and other forum members are more interested in boosting their ego by showing their depth of knowledge on the topic (and correcting others on the topic), whereas corporate websites are more interested in raking profit by displaying (potentially) dishonest information.
Google search used to have a "discussions" tab until they killed the feature a few years back. It surfaced forum content with real peoples' opinions.
There are opportunities for domain-specific search engines (e.g. software engineering, academic research, products). Search verticals that are hand-curated would be very interesting to see.
One idea to avoid SEO spam is to let people vote on content they trust or flag spammy results. Some search engines are experimenting with customized filters for non-commercial sources or user-generated content like Reddit. One concern about this is that many customization options result in more friction for users in comparison to Google's simple input box.
DDG offers a great alternative to surveillance engines like Google, but being mostly based on Bing's search engine doesn't create a quantum leap in better search results.
Paid search engines could avoid the conflict of interest between providing the most relevant results and satisfying advertisers. Even Larry Page and Sergey Brin have admitted this challenge in a research paper : "The goals of the advertising business model do not always correspond to providing quality search to users. ..... we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers"
There is a market for a paid search engine, but the paid search engine needs to be based on a self-owned and -operated index. Many Google alternatives like DDG use the Bing search API, which makes them dependent on a big search provider.
With the rise of very powerful generative language models like GPT-3 or PaLM, the way search engines work is changing. In the future, they will be more focused on generating results instead of finding results that are already out there. This shift will have a major impact on the way information is gathered and distributed.