In June 2021 GitHub featured a singular tool. An assistant capable of autocomplete and generate code thanks to artificial intelligence. They called it GitHub Copilot. At that time, the unknown was clear: can an AI really program? The doubts were many, especially considering that the limited launch of GPT-3 in 2020 had hardly attracted attention.
However, Copilot, which was precisely based on said artificial intelligence model, showed that things had changed. This service became a revolution for the developers in a very short time, who verified how, thanks to this component, it was possible to save a lot of time in repetitive tasks. Suddenly machines could program (somewhat) for us.
The impact of this assistant was enormous: in a few months it “chopped” 30% of the new code that was published on GitHub, and shortly after that quota rose to 40%. The forecast? That in five years it reaches 80%.
This advance has been overshadowed by ChatGPT, the engine that OpenAI presented in November 2022 and which posed a radical change not only for developers, but for all kinds of professionals and users. Its application was much more general and was not confined to the world of programming, but even so it was also possible to use it for programming.
Progress in this area has been —as many of you have seen— frantic. A few days ago, GPT-4 was presented, the new conversational artificial intelligence model that adds important new features (it is multimodal and supports text but also image input) and above all it is more powerful and precise. In a few days we have seen how amazing use cases have emerged for this new AI engine, and among them is programming.
Copilot, ChatGPT and GPT-4, faced
Precisely that capacity and those improvements proposed in GPT-4 have made us wonder if this model changes things even more for programmers. And to answer that question, we wanted to ask two experts in this field, who have given us their opinion on these platforms, their scope and, of course, their future.
The first of the experts we wanted to turn to to answer these questions is Brais Moure ( @MoureDev ), a software engineer who has been a freelancer specializing in mobile development for eight years and who has also been a disseminator and creator of programming content on their YouTube or Twitch channels.
Brais confessed to us how the appearance of GitHub Copilot surprised him like everyone else. “We turned to try it and see how his responses were in different languages and attending to each case.” This expert has continued to use it from time to time, but confesses that “usually not.”
For him Copilot “is a great tool” that allows him to answer some questions that “before I was looking for in Google/Stack Overflow/GitHub” or “generate that first code to continue completing it myself.” However, for him, the famous quote from Spider-Man would apply here: ” with great power comes great responsibility .” In fact, he explains the following:
“Unloading all our responsibility as programmers on a tool like Copilot can lead to losing control of our software. In summary, first of all, we must understand that this automatic response is the one that suits us (and have the ability to adapt it to our needs of project). In other words, it is the one that we would arrive at, but perhaps taking more time”.
There is one more important detail about Copilot. As Brais commented, “the best thing about Copilot is that it’s integrated into our code editor” and is expressly created for this area, something that makes it especially powerful for developers.
The second of our experts is Miguel Ángel Durán ( @midudev ), a programmer with more than 15 years of experience. Like Brais, he is also a content creator and posts his content to both YouTube and Twitch, and he matches that vision of the usefulness of the GitHub-created tool.
Miguel Ángel told us how he has used Copilot “since day one”. Thanks to him, he points out, “my productivity has improved a lot. I like it because it integrates perfectly with my workflow. It is not intrusive because it is like advice.”
For him Copilot is a particularly useful assistant. “For example, if I don’t know how to come up with a solution at a given moment, your suggestion, even if it’s not the best, helps me think in a direction. I still don’t even use your suggestion afterwards, but it unlocks me to continue. ” There is another even more essential advantage: saving time that you can spend on other tasks:
“Other times it helps me with repetitive tasks. How many times have I written a loop to extract a field from an array of objects? Many. GitHub Copilot gets me writing it in a second and lets me focus on what really matters.”
When ChatGPT came out, both experts also spent time testing what this AI engine could bring to their work. For Brais, ChatGPT has a more general orientation: “you can practically present the complete use case to ChatGPT (and iterate on the responses), since it is a conversational model”. For him the feeling is that GPT is “smarter, if I may use the expression. I think it gives me better answers, especially if you make things difficult for it.”
Miguel Ángel has also used ChatGPT in his workflow, but above all “to answer emails or create documentation. I have been able to ask him some programming questions, but it is heavier for me than the immediacy of GitHub Copilot. The difference is that Copilot is more specific to programming and also uses the context of what you’re already creating, it also tends to suggest a single line of code instead of entire blocks.
As Brais already pointed out, Miguel Ángel believes that “ChatGPT is more general, it gives you larger blocks of code and with explanations”. For him ChatGPT is the best option if you need to learn something or you are not sure how to do something because you lack the knowledge.
We were especially interested in hearing about your first impressions of GPT-4, OpenAI’s new conversational AI model. For Brais “it runs much slower than version 3″, something that the company already warned in the official announcement, but in return its results ” are more detailed or consistent, but not always better, more efficient or “elegant” than the version 3.5 (especially when generating specific code)”.
Not only that, says Brais: version 4 does everything in 3.5 “and shows me step by step and with code how to do things.” Furthermore, he explained, GPT-4 is multimodal and also recognizes images as input, something that “opens up infinite possibilities for us.”
That was also the perception of Miguel Ángel, who explained to us how after trying it for a few hours he found it more powerful. ” He is more precise and smarter, although slower.” He also pointed out something that Brais had also had an impact on: “in ChatGPT the code solutions are normally tried to be grouped into a single block. On the other hand, GPT-4 gives you the solution step by step and explains each one. It is more didactic and more iterative in its solutions”.
Putting CoPilot, ChatGPT and GPT-4 to the test
We asked both Brais and Miguel Ángel for a special favor: put these tools to the test to see how they solved a specific problem. Brais actually wanted to do two tests. The first, with a program that calculates who wins the most games of rock, paper, scissors, lizard, Spock (great!). The prompt was the following:
Create a program that calculates who wins the most games of rock, paper, scissors, lizard, spock.
The result can be: “Player 1”, “Player 2”, “Tie” (tie)
The function receives a list containing pairs, representing each move.
The pair can contain combinations of “????” (rock), “????” (paper),”✂️” (scissors), “????” (lizard), or “????” (spock).
The test was part of the programming challenges that Brais poses to his followers, and after proposing this specific problem, his conclusions were the following:
- GPT-4 : The solution was less concise, but easier to understand.
- GPT-3.5 : It was more concise, but more complex.
- Copilot – After giving it a couple of spins, it hasn’t been able to figure it out.
In fact, after the test it confirmed how Copilot “has a more oriented approach to solving everyday tasks based on similar code already published.” It’s something that was already known, but GitHub Copilot basically does a copy/paste of something that has already been done literally by someone else. The GitHub wizard “is worse at understanding a complex premise.”
This expert also gave him another example: that of creating “a regular expression in python to find out if the IBAN of a bank account is valid.” After analyzing the result, Brais concluded that:
- GPT-4 : It has adhered to the request, it has implemented a regular expression, but it has clarified that it may not be enough to validate the IBAN.
- GPT-3.5 : Has implemented a regular expression and additional validation to 100% validate the IBAN.
- Copilot : You haven’t implemented a regular expression, but you have a correct way to validate it.
For this programmer, the best solution was that of GPT-4 “because of its way of understanding the context, explaining it and dealing with it”. Of course, he warned that this does not mean that at the code level it is the “best”.
Miguel Ángel also had the same detail and invested time in evaluating the behavior of these three platforms when solving a programming problem. Specifically, he used the “Create Super Mario Bros. game with JavaScript” prompt.
As in the case of the issues raised by Brais, Miguel Ángel confirmed that GitHub Copilot is not so intended to solve entire problems, and “is more intended to give you suggestions of your own code. The more code you have in the file or the project gives better results.”
With GPT-3.5 (the ChatGPT which is freely available on the OpenAI website ) he checked how the code was “a simplified version”, in a single block and “a very general explanation of the code”.
The result, he explained, works, “but it is very far from having any reference to the platform game. It is something tremendously basic. He has been very quick to respond, yes.”
With GPT-4 things changed a lot. “Separa separates all the files and their explanations, instead of doing it all in one block. What’s more, everything it does and the practices it uses are more advanced,” Miguel Ángel explained to us.
Not only that: in that version of the OpenAI AI engine, the code was also more clearly segmented: “it gave me 5 sections: The Basic HTML, JavaScript File Configuration, Player Class Creation, Keyboard Control, and Game Loop.” game”. In fact, he noted, it even implements the “characteristic blue background and collisions.”
What Michelangelo did next is copy the code and change the image to a Mario sprite. After doing it, this basic game appeared but it allows me to control Mario, jump, has gravity and collisions with the sides. Of course, it looks much more advanced and complete than ChatGPT. The image is very large, because it is The first one I’ve come across, but with a few tweaks it would have a recognizable Super Mario Bros. base.”
Of course, there was another important difference regarding the behavior of GPT-4 compared to its predecessor. “The ChatGPT version took only 3 or 4 seconds. The GPT-4 version took much longer to produce it, almost a minute, and I also had to tell it to repeat the last point because the code didn’t tell me.” full dice”. For Miguel Ángel, the latter is normal since GPT-4 has just made its appearance and it will surely improve significantly in the short term.
The conclusions of both developers were forceful. According to Brais, GPT-4 ” we must take it as a help tool, not as a source of universal truths. And, much less, as a threat. If I have to get wet”, he concluded, “I stay with GPT before with Copilot, and with version 4 before with 3.5 (ChatGPT)”.
For Miguel Ángel “GitHub Copilot offers a perfect development experience if you’re already in the editor. I plan to continue using it and I’m not going to change it to ChatGPT/GPT-4”. Here the idea is to use these new models almost more for other areas —pointing to a substitute for the Google search engine—, and he pointed out that “if you are going to get performance out of it”, paying the 20 dollars a month that it costs to use GPT-4 “can worth”.
