Human or AI, which is the best coder?
Over the last couple of months, the office has been abuzz with conversations of AI-generated code. Our slack was flooded with messages from engineers volunteering to test drive a corporate license of Copilot. I’ve been quietly evaluating Copilot, and after having tried ChatGPT and Bing, today I got access to Bard.
I decided to use each AI to solve a problem I was confident I could solve pretty easily and that the AIs would have no problem doing.
Here is the problem exactly as I submitted it to the AIs:
In typescript, write a binary search function that takes in an array of a generic type, a target of the same type, a compare function for those types. Return the index of the target if found, or the negative value of the index where the target should be inserted.
I would then ask a follow up questions for the AI:
Write an example using an array of sorted integers
Write an example using a type Person
The human solution
function binSearch<T>(
arr: Array<T>,
target: T,
compare: (a: T, b: T) => number
): number {
//
let i = 0;
let j = arr.length - 1;
let mid;
while (i <= j) {
mid = Math.floor((i + j) / 2);
if (compare(target, arr[mid]) < 0) {
// target is in first half
j = mid - 1;
} else if (compare(target, arr[mid]) > 0) {
// target is in second half
i = mid + 1;
} else if (compare(target, arr[mid]) === 0) {
// found it
return mid;
}
}
return -i;
}
const compareNumbers = (a: number, b: number) => a - b;
console.log(binSearch([0, 1, 2, 3, 4, 5], 3, compareNumbers)); // expect 3
console.log(binSearch([0, 1, 2, 4, 5], 3, compareNumbers)); // expect -3
type Person = { name: string; age: number };
const comparePersonByAge = (a: Person, b: Person) => a.age - b.age;
console.log(
binSearch(
[
{ name: "Alice", age: 10 },
{ name: "Bob", age: 20 },
{ name: "Charles", age: 30 },
],
{ name: "Bob", age: 20 },
comparePersonByAge
)
); // expect 1
console.log(
binSearch(
[
{ name: "Alice", age: 10 },
{ name: "Charles", age: 30 },
],
{ name: "Bob", age: 20 },
comparePersonByAge
)
); // expect -1Let’s see how the AIs did:
Copilot
Here is what Copilot produced:
function binSearch<T>(
array: T[],
target: T,
compare: (a: T, b: T) => number
): number {
let min = 0;
let max = array.length - 1;
while (min <= max) {
const mid = (min + max) >> 1;
const comp = compare(array[mid], target);
if (comp < 0) {
min = mid + 1;
} else if (comp > 0) {
max = mid - 1;
} else {
return mid;
}
}
return ~min;
}It’s pretty much the same, except instead of returning the negative index, it returns the complement of the integer minus one (the correct negative index to insert the target into the array).
However, Copilot shined in that I could easily use it from my IDE, and it made writing examples very quick as I just typed a command and the editor did the rest:
// example of binSearch usage
const array = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
const index = binSearch(array, 5, (a, b) => a - b);
console.log(index); // 4
// example of binSearch usage with custom compare function
const array = [
{ id: 1, name: "John" },
{ id: 2, name: "Jane" },
{ id: 3, name: "Jack" },
{ id: 4, name: "Jill" },
{ id: 5, name: "Joe" },
];
const index = binSearch(array, 3, (a, b) => a.id - b);
console.log(index); // 2
// example of binSearch usage where the target is not in the array
const array = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
const index = binSearch(array, 11, (a, b) => a - b);
console.log(index); // -11ChatGPT
ChatGPT’s solution:
function binarySearch<T>(
arr: T[],
target: T,
compare: (a: T, b: T) => number
): number {
let left = 0;
let right = arr.length - 1;
while (left <= right) {
const mid = Math.floor((left + right) / 2);
const comparison = compare(arr[mid], target);
if (comparison === 0) {
return mid; // target found at index mid
} else if (comparison < 0) {
left = mid + 1;
} else {
right = mid - 1;
}
}
// target not found, return negative value of index where target should be inserted
return -left;
}
// example 1
const arr = [1, 3, 5, 7, 9];
const target = 5;
const compare = (a: number, b: number) => a - b; // compare function for numbers
const index = binarySearch(arr, target, compare);
if (index >= 0) {
console.log(`Target found at index ${index}`);
} else {
console.log(`Target not found. Insert at index ${-index}`);
}
// example 2
type Person = {
name: string;
age: number;
};
const people: Person[] = [
{ name: "Alice", age: 25 },
{ name: "Bob", age: 30 },
{ name: "Charlie", age: 35 },
{ name: "David", age: 40 },
{ name: "Eve", age: 45 },
];
const target: Person = { name: "Charlie", age: 35 };
const comparePeople = (a: Person, b: Person) => a.age - b.age;
const index = binarySearch(people, target, comparePeople);
if (index >= 0) {
console.log(`Target found at index ${index}`);
} else {
console.log(`Target not found. Insert at index ${-index}`);
}ChatGPT did really well, the solution came with explanations (which is not the case for Copilot), and it was easy to ask the AI to do more, like generate the examples.
Bing
function binarySearch<T>(
arr: T[],
target: T,
compareFn: (a: T, b: T) => number
): number {
let left = 0;
let right = arr.length - 1;
while (left <= right) {
const mid = Math.floor((left + right) / 2);
const cmp = compareFn(target, arr[mid]);
if (cmp === 0) {
return mid;
} else if (cmp < 0) {
right = mid - 1;
} else {
left = mid + 1;
}
}
return -(left + 1);
}const arr = [1, 3, 5, 7, 9];
const target = 5;
function compareFn(a: number, b: number): number {
return a - b;
}
const index = binarySearch(arr, target, compareFn);
console.log(index); // 2interface Person {
name: string;
age: number;
}
const people: Person[] = [
{ name: "Alice", age: 25 },
{ name: "Bob", age: 30 },
{ name: "Charlie", age: 35 },
{ name: "David", age: 40 },
];
function comparePeople(a: Person, b: Person): number {
return a.age - b.age;
}
const targetPerson: Person = { name: "Charlie", age: 35 };
const index = binarySearch(people, targetPerson, comparePeople);
if (index >= 0) {
console.log(`Found ${targetPerson.name} at index ${index}`);
} else {
console.log(`Insert ${targetPerson.name} at index ${-(index + 1)}`);
}Obviously, since Bing uses ChatGPT, the results are very similar.
Again, Bing output a bunch of useful comments and explanation.
Bard
Finally we’re going to use Google’s Bard, and the results are… surprising.
All my attempts were rejected with various error messages, so I kept simplifying the request. Alas, still no code.
Maybe Bard doesn’t speak TypeScript, so let’s try JavaScript:
Still, Bard doesn’t know what to do. But I did get some sort of result when not specifying a language:
This pseucode very much looks like JavaScript that would do the job (here, — 1 means not found), but it’s pretty disappointing for a long awaited AI chatbot which has already fell short in the past.
Conclusion
As much as I’d like to say my code was the best, it was not. It was at best identical or on par with the AIs. Even if the AIs produced something slightly different, it’s still close enough that I can tweak it in a few seconds to get the expected result, and my code certainly took the longest to write. See, even though I had the algorithm down, I made a silly mistake, I forgot to increment and decrement mid when reassigning it the start/end values. It took me a little longer than I care to admit to notice my mistake, one the AIs would never make. My take away is that I can use the AI to do most of the heavy lifting, and tweak the final result. I think of it like a construction worker using a paint roller to do the large surfaces and a fine brush for the small details.
I plan on continuing using Copilot for its ease of use and integration in VSCode, and ChatGPT/Bing for the bigger stuff. As for Bard though, it’s clearly not ready for developers.
