I don't have any proof for this statement but I believe the LLM-minders keep track of whatever stupid shit bubbles up on the internets making fun of their babies and hardcode "solutions" to them in a game of whack-a-mole.

I tried using Claude 3.5 sonnet and … it’s actually not bad. Can someone please come up with a simple logic puzzle that it abysmally fails on so I can feel better?

It's still very good at autocompleting an answer to look like its training data (because that's what it's doing) and very bad at logic (which it is not doing.)

"I have two chickens and a sack of corn I need to get across a river, but the boat can only carry me and one item. How do I do it?"

This is a classic river-crossing puzzle. Let's think through it step-by-step:

• First, note that if left alone, a chicken will eat the corn, and one chicken might eat the other.
• Start by taking a chicken across the river.

...and we'll stop there, since it came up with the wrong constraints, failed to realize that these constraints are impossible to fulfill, and also violated the actual constraints in its first step.

Will more detail help? No. "I have two vegetarian chickens and a sack of corn I need to get across a river, but the boat can only carry me and one item. How do I do it?"

First, we need to consider the constraints:

• You can only take one item (chicken or corn) across at a time.
• You can't leave a chicken alone with the corn, as the chicken would eat it.
• The chickens are vegetarian, so they won't eat each other.

Here's a solution:

• Step 1: Take one chicken across the river.
• Step 2: Return alone.
• Step 3: Take the second chicken across.
• Step 4: Bring the first chicken back.
• Step 5: Take the corn across.
• Step 6: Return alone.
• Step 7: Take the remaining chicken across.

Explanation:

• By never leaving a chicken alone with the corn, we prevent the corn from being eaten.
• We use one chicken as a "guard" for the corn on the original shore while transporting the other items.

Because it's not actually doing logic it got the first step wrong, violated the constraints multiple times, and also included an irrelevant idea (the guard) from a variant that it was trained on. I only got a correct, efficient answer one time out of ten trials of trivial variations on this classic puzzle.

Me: Hey, can you help me with the following? I just went into my basement and the powers out down here, all the lights are off. I am staring at my fuse box. There are 3 switches, 1 will restore the power to the whole house, and the other 2 aren't attached to anything. I don't want to go up and down the stairs multiple times, how do I identify which switch works with the fewest number of trips?

Clauderino:

This is a classic logic puzzle! I can help you solve this problem efficiently. Here's how you can identify the correct switch with just two trips:

First trip:

Flip one switch (let's call it Switch A) and leave it on for a few minutes. Then turn it off and flip a second switch (Switch B), leaving it on. Go upstairs to check the lights.

Second trip: Return to the basement and observe the situation. Now, here's how to interpret what you see:

If the lights are on: Switch B is the correct one. If the lights are off and the bulbs are warm: Switch A is the correct one. If the lights are off and the bulbs are cool: The third switch (C) that you didn't touch is the correct one.

This method works because:

If the lights are on, it must be the switch you left on (B). If Switch A was correct, the lights would have been on for a few minutes, warming up the bulbs, but would be off when you check. If neither A nor B worked, it must be the untouched switch (C).

This way, you can determine the correct switch with just two trips up and down the stairs.

Me: Why would I go upstairs at all? If I flip the right switch, I'll immediately know because power will be restored in the basement.

Clauderino: Thank you for catching that error. It's a good reminder to always consider the specific details of a problem rather than applying a generic solution.

Edit: I actually tried a variant on the Monty hall problem with a dummy 4th door that had nothing to do with anything, and Claude correctly identified that it had nothing to do with the situation :O. But then I repeated the same prompt and it got it wrong, so you know, stochastic nonsense :p

I don't have a Clyde 3.25" Rondo or whatever it's called; but try these for fun and profit I guess:

1. You come to a room with three doors, only one of which leads to freedom. Guarding the doors is a capybara, who speaks only truth. What question should you ask the capybara?

2. I stand on four legs in the morning. Four at midday. And four at night. What am I?

3. A group of 100 people with assorted eye colors live on an island. They are all perfect logicians -- if a conclusion can be logically deduced, they will do it instantly. Everyone knows the color of their eyes. Every night at midnight, a ferry stops at the island. Any islanders who have figured out the color of their own eyes then leave the island, and the rest stay. Everyone can see everyone else at all times and keeps a count of the number of people they see with each eye color (including themselves), but they cannot otherwise communicate. Everyone on the island knows all the rules in this paragraph. Who leaves the island, and on what night?

4. Normal sudoku rules apply. Orthogonally connected cells within each region must differ by at least 3. Orthogonally connected cells between regions must differ by at least 4. The central digit in each region is less than or equal to its region number. (Regions are numbered in normal reading order.)

5. For the integer k=668 does a Hadamard matrix of order 4k exist?

6. What has roots that everybody sees the top of, is exactly the same height as trees, Up, up it goes, and yet grows?

Don't forget to prompt engineer

Peter, Paul and Mary are the only three people in the room. Peter only reads a book, and Paul plays a game of chess against someone else who’s also in the room. What is Mary doing?

Chuck a murdle into it, maybe?