Alright, I finished writing my bot, @ClaudeSonnet3539. It uses (new) Claude Sonnet 3.5 on the backend, and its prompt is very similar to the one from FiveThirtyNine.

The bot traded in 913 markets - it couldn't trade in all 1000, due to a bug in the market selection code that resulted in 75 duplicate markets as well as some markets that couldn't be traded in for various reasons. When scoring the bot, I'll use the logic described above, for M = 913:

...I'll scale the actual profit by 1000/M. For instance, if I could only find 800 markets, and then the actual profit made by the highest-performing bot is Ṁ40, I would resolve the market to 40*(1000/800) = 50%.

Later, I'm going to clean up my code and upload it, along with a complete transcript the bot's reasoning for all 913 markets. I'll also post another comment with an explanation of how the bot works.

I'm feeling pretty good about the bot's performance - most of its reasoning seems pretty astute, and its given probabilities are generally quite close to those of the market, despite not being told the market's probability in its prompt. I'm looking forward to seeing its profits in a month!

On December 1, 2024, I will select 1,000 random YES/NO markets on Manifold that will resolve in about one month (see "Selecting markets" for more information). I will post a .txt file containing links to the 1,000 markets in the comments of this market.

@traders The challenge begins: https://raw.githubusercontent.com/CDBiddulph/motley-bot-resources/refs/heads/main/select_markets/output/competition_markets.txt

Hopefully I can make the time today to set up a Claude bot to enter the contest. No promises

One idea maybe worth exploring would be phrasing predictions about the world such that they end up about predicting a token.

For instance: "From this available data before the election it could already have been deduced that the USA president would eventually become ..."

So you like meta prediction markets? Well have I got a treat for you! https://manifold.markets/CKLorentzen/what-will-be-the-topic-of-the-highe

@traders Just upgraded this market to Crystal! And now, I've written a Basic Bot for you to start out with, so you can easily create your perfect bot in this final month of the challenge!

I created this market with a whole year of lead time and it was quite popular then - at least for the traders. I probably gave you all too much time - people weren't feeling the time pressure, so AFAIK they didn't actually start writing any bots 😅. But with only a month until the challenge begins, now is the time to start creating those bots in earnest!

I created a brand-new Basic Bot that everyone can start out with - it handles reading in the markets, writing a basic prompt for an LLM with all the relevant information, and making trades based on the LLM's predictions. The only thing it doesn't do is the fun part - running an LLM to make those predictions!

I'm hoping to see the next great AI forecaster make its debut appearance here. Please let me know if you're planning to participate!

You could really do something as simple as taking my basic bot code, calling your favorite LLM with no other context than the market info and Bing search results, and parsing BUY_YES, BUY_NO, or DO_NOTHING from the response - writing this code could probably take you an hour or less. If no one else does this, I will, so take the glory before I do!

I'm also willing to pay up to ~$50 per entry for LLM API costs, if the cost will be a concern for you. DM me if you're interested

With the real money coming into play, I suspect this will be impossible because of the flat 1 bet.

You're not letting bots weight the markets by confidence, which would be their important advantage.

@gpt_news_headlines The idea is that the best strategy is to have an opinion about every market, rather than developing a really good strategy for just a subset of markets. This is supposed to enforce that some common-sense reasoning about the world comes into play rather than just patterns in the market itself, i.e. the bots will have to use something like an LLM.

@gpt_news_headlines I'm not sure if or how the real money markets will affect bots. If all else fails, we can keep track of the bots' "hypothetical bets" on a spreadsheet and calculate how much they would have profited if they had been able to bet.

If a bunch of bot authors participate in the competition, it is very likely that at least one bot will make a M100 profit simply due to luck, even if all bots trade randomly.

@lumi My math could be wrong, but I think that would be very unlikely.

With the assumption that a bot bets in every market, and every market is at 50% (so it either loses or doubles its money), the bot would have to guess correctly for 550 markets. Using the binomial distribution, the probability of a single bot doing that if it randomly guesses is 0.07%. To get to a 50% probability of a single random win, 990 bots would have to participate.

I address this general concern in the paragraph starting, "As more bots participate in this challenge..."

1000 markets is a lot especially if someone plans to use GPT-4 API it could add up to $100 or more. I think making 3 smaller rounds with 100 markets each with disqualification would be more entertaining and solve a lot of problems...

@patrik Good point, I'll think about how to take into account the cost of LLM APIs. I don't want to entirely cancel the 1,000-market contest in favor of smaller contests, but I think smaller rounds would be interesting.

I boosted this market (https://manifold.markets/CDBiddulph/what-will-be-the-score-of-the-bot-t), as recommended by this market (https://manifold.markets/CDBiddulph/how-much-would-these-interventions).

The main market is currently trading at 68%, and the market predicts it'll be at 71% in two weeks. I'm not sure I believe this, but feel free to take that into account (and possibly correct for anyone trying to manipulate the main market).

🚨 Free Money Alert 🚨

According to the market here, this market's probability in 3 weeks is going to be 4-7 percentage points higher than its probability in 1 week! You should either bet this market higher now in anticipation of that jump, or bet down the predictions in that market: https://manifold.markets/CDBiddulph/how-much-would-these-interventions?r=Q0RCaWRkdWxwaA

I just finished the script I'll be using to select the 1,000 markets: https://github.com/CDBiddulph/motley-bot-resources/

I ran the script for each month of the year until the start of 2025 and included the results in the repo. You might want to use these files to validate your bot's performance.

Although no month before January 2025 has 1,000 eligible markets that close at the start of the month (even with relaxed filtering), it looks like there are already enough eligible markets that close around January 1, 2025 (1,515 markets with the strictest filtering, 6,411 with relaxed filtering). In addition to the file with just 1,000 markets, I included a file with all 6,411 markets that close around January 1, ordered by filtering priority.

I might start holding preliminary challenges at the start of each month once we have at least two contenders, e.g. competing on 100 markets that close around July 1 at the beginning of June.

What if I release a source code which shows that the bot places bets randomly, but place all the bets myself by running a separate script?

@patrik Or just use an LLM since that's random enough.

@patrik IIUC that wouldn't count, if the open-sourced code isn't the actual code you run? See my suggestion below for only open-sourcing the code to registered participants in the market.

I'm also open to having everything be closed-source until after the contest, though I think open-source has its advantages. I hadn't thought about how people could be incentivized by the exclusive right to use their code to make profits on non-eligible markets.

@CDBiddulph Yeah but how will you be able to tell that the code hasn't been run when it relies on randomness?

@patrik Uh I guess I would just trust you not to cheat 😛 If there was suspicion I could always rerun your code and see if the profit is way lower

@CDBiddulph You could also add one more round with execution by third party for the top 5 winners or something.

@patrik Maybe I will do something like that. It's a bit of a hassle, but maybe not too bad

"The nice thing about prediction markets is that, assuming everyone accurately assesses the quality of their own work and trades accordingly, someone who builds off of someone else's work only profits on whatever additional value they created."

Not true. If you manage to place bets faster with otherwise completely equal strategy, then you are technically robbing the person of the profits.

Therefore, this competition alone may not be enough incentive to make the bot open-source.

Yes, for this competition it doesn't matter, since I can place the bets and then make it open-source, but if I do, I won't be able to make profits from it later.

@patrik I think the way I wrote that was confusing. What I meant was that you can make a profit off of this market by betting it higher and then announcing that you built a high-performing bot.

As for making profit off of the Eo2024 markets, I'm asking that no one use bots to bet on these markets until the contest starts:

Hypothetically, someone could use someone else's code to trade on a bunch of eligible markets ahead of time, wiping out whatever alpha their bot might have. I don't think this will be a problem, but please don't do this!

I'll change this to be clearer that you also shouldn't use your own bot to trade on eligible markets until the challenge starts. You can use the testing version of your bot to trade on any non-eligible markets you want and possibly profit off of them.

Thinking about it, it might be a good idea to only open-source the bots to people who have registered their participation in the contest. That way it's a lot easier to track who might be illicitly using bot code to make trades on Eo2024 markets.

@CDBiddulph Oh ok. Still I'm not sure if people will be willing to open-source it, because not doing so may generate more profit in the long run (if no one steals my ideas).

@patrik I would like to make it a hard requirement that the code eventually gets open-sourced, since my goal with this challenge to expand humanity's collective predictive ability.

But if going open-source at the end of the year would be a dealbreaker, maybe adding in a guaranteed prize pool for the highest-scoring bot would help. (This only makes sense if bots are closed-source until Eo2024, since otherwise credit-attribution would be hard.)

Let me know if you have any ideas - or add them to this market and you'll get 50 mana if I use them: https://manifold.markets/CDBiddulph/how-much-would-these-interventions?r=Q0RCaWRkdWxwaA