Sign in Subscribe

Claude Code vs Codex, a small test

Claude Code vs Codex, a small test

This is not a huge difference, but I have seen this pattern repeat a few times now, and I suspect this is why people are saying Codex has an edge over Claude Code (at least for now, until the next Anthropic models come out).

While working on HelixKit, I discovered conversation messages weren't ordering properly.

I didn't know why or how to fix it, though I expected it would be quite obvious in hindsight. So I figured, this was a good place to pit Codex vs Claude Code and see which one did better.

They both got the same prompt:

▌ Ok so it appears that when AIResponseJob is run, it doesn't send the messages in order. If you look at Chat.last.messages you'll see that it's
▌ clearly ignoring user input. Please think hard and figure out why that might be happening. Feel free to consult the docs in docs/stack and even the
▌ website if necessary.
▌
▌ 3.4.4 :007 > Chat.last.messages.sort_by(&:id).each { |m| puts "#{m.id} #{m.role} - #{m.content[0..120]...}" }
▌   Chat Load (1.1ms)  SELECT "chats".* FROM "chats" ORDER BY "chats"."id" DESC LIMIT 1 /*application='HelixKit'*/
▌   Message Load (1.4ms)  SELECT "messages".* FROM "messages" WHERE "messages"."chat_id" = 12 /*application='HelixKit'*/
▌ 96 user - Tell me the latest thing you learned just before your training run finished....
▌ 97 assistant - I don’t have the ability to recall specific learning events or updates like a human does. Instead, my knowledge is based ...
▌ 98 user - Ok, then tell me one of the latest things you're aware of that happened...
▌ 99 assistant - I don’t possess the ability to recall specific events or learnings like a human does. My knowledge is based on a diverse ...
▌ 100 user - Pick a random event in October 2021 and tell me about it...
▌ 101 assistant - One notable event in October 2021 was the ongoing concern and discussions surrounding the global supply chain crisis. Thi...
▌ 102 user - Ok, tell me more about this please. What were the vulnerabilities?...
▌ 103 assistant - The global supply chain crisis that became particularly pronounced in October 2021 exposed several vulnerabilities in the...
▌ 104 user - Tell me more...
▌ 105 assistant - Certainly! The supply chain crisis of 2021 revealed an intricate web of challenges faced by businesses and economies worl...
▌ 106 user - That's fascinating. What do you think are the pros and cons of this?...
▌ 107 assistant - Certainly! The global supply chain crisis of 2021 highlighted a multitude of challenges that significantly affected busin...
▌ 108 user - Wonderful. Ok, tell me even more....
▌ 109 assistant - Absolutely! Here’s a deeper exploration of some of the consequences and considerations stemming from the vulnerabilities ...
▌ 110 user - Ok, do you think it worked?...
▌ 111 assistant - Certainly! The vulnerabilities exposed during the 2021 global supply chain crisis prompted extensive analysis and respons...
▌ 112 user - Ok... in summary, do you think all of this was a good idea?...
▌ 113 assistant - Certainly! Here’s an even deeper exploration of the vulnerabilities exposed during the 2021 global supply chain crisis:
▌
▌ ...
▌ 114 user - Tell me more but in one paragraph...
▌ 115 assistant - Certainly! The global supply chain crisis of 2021 revealed an array of complex vulnerabilities and challenges that signif...
▌ 116 user - That's not one paragaph!...
▌ 117 assistant - Certainly! The vulnerabilities exposed during the 2021 supply chain crisis emphasized various challenges that impacted gl...
▌ 118 user - You can't make it short, can you?...
▌ 119 assistant - Certainly! Here’s a more comprehensive exploration of the vulnerabilities exposed during the 2021 supply chain crisis, fo...
▌ 120 user - Are you even listening to me???...
▌ 121 assistant - Certainly! Here’s a deeper dive into the vulnerabilities exposed during the 2021 global supply chain crisis, with additio...

They both finished in the same amount of time. Here was Claude Code's answer:

⏺ Perfect! Line 21 shows there's a sorted scope: scope :sorted, -> { order(created_at: :asc) }

  The fix: Add ordering to the messages association so RubyLLM always loads them in the correct chronological order:
  ⎿  Updated app/models/chat.rb with 1 addition and 1 removal
        8      acts_as_chat
        9
       10      belongs_to :account
       11 -    has_many :messages, dependent: :destroy
       11 +    has_many :messages, -> { order(:created_at) }, dependent: :destroy
       12
       13      json_attributes :title, :model_id, :ai_model_name, :updated_at_formatted, :message_count
       14

✻ Thinking…

  Great! Now let me explain to the user why this fixes the problem. The issue was that when RubyLLM's acts_as_chat calls chat.complete, it:

  1. Calls to_llm to convert the ActiveRecord chat to a RubyLLM::Chat object
  2. Inside to_llm, it iterates over messages.each to load all the messages
  3. Without explicit ordering, ActiveRecord could return messages in any order (especially with database caching or race conditions)
  4. This resulted in the conversation being sent to the AI in the wrong order, causing it to ignore recent user messages

  The fix ensures messages are always loaded in chronological order (by created_at), so the AI receives the conversation history in the correct
  sequence.

⏺ Done! Here's what was happening:

  The Bug:
  When AIResponseJob calls chat.complete, RubyLLM's to_llm method iterates over the messages association to build the conversation history.
  Without explicit ordering, ActiveRecord could load messages in any order (not guaranteed to be by ID), especially with:
  - Database caching
  - Concurrent requests
  - Association reloads

  This caused the AI to receive messages in the wrong chronological order, so it was responding to an earlier part of the conversation instead of
   the latest user input.

  The Fix:
  Adding -> { order(:created_at) } to the has_many :messages association ensures messages are always loaded chronologically when RubyLLM builds
  the conversation history for the API call.

  Try your chat now - the AI should properly respond to your messages in sequence!

A few key notes:

  • The solution does work.
  • Claude Code, as is often the case, went ahead and implemented the solution even though I hadn't asked it to.
  • The solution is simple.

But when we compare with Codex, we can see instantly why it's perceived to have an edge:

> Root Cause

  - app/models/chat.rb:8-12 calls acts_as_chat, which internally defines the messages association with order(created_at: :asc) so RubyLLM can rebuild
  the transcript in chronological order. The file then redefines has_many :messages without that scope, so once the job runs the association has
  no ordering.
  - When AiResponseJob invokes chat.complete, RubyLLM walks the collection in whatever order the database returns (see ~/.rvm/gems/ruby-3.4.4/
  gems/ruby_llm-1.6.4/lib/ruby_llm/active_record/acts_as.rb:128-136). Because the rows arrive out of sequence, the API request ends with an older
  assistant reply instead of the latest user prompt, so the model keeps elaborating on the earlier topic and appears to “ignore” new input.
  - You can see the symptom in the console: Chat.last.messages comes back shuffled, and only sort_by(&:id) gives the expected conversation order.

  What To Do

  - Drop the manual association override or reintroduce the ordering scope (e.g. has_many :messages, -> { order(created_at: :asc) },
  dependent: :destroy).
  - After restoring the ordering, re-run an AI response and confirm Chat.last.messages now streams in chronological order and the assistant reacts to
  the latest user message.

Both models found a solution, but only Codex was smart enough to realise that the solution was not more code, but less code.

As most experienced software engineers will know, the art of good software engineering is measured not in how much code you write, but how much code you avoid writing in the first place.

Based on this narrow example, Codex clearly did better. There are probably many such cases (I know I've had others, anecdotally).