MIT analysized ChatGPT test for improvement of our productivity

chatgpt by openai

The recent introduction of ChatGPT and other similar artificial intelligence (AI) tools has brought many novelties to the world, both personal, social and work due to its great capabilities. And a study here shows whether or not the results are good when the tool is used properly.

We have already seen the history of teachers who, instead of positioning themselves against the AI ​​of the OpenAI company, have decided to adopt it as an ally for learning. It must be remembered that a major failure of ChatGPT is that it can generate texts that sound very convincing but include inventions, and that is that this text generator does not indicate the sources of what it says or the information collected is old.

This new study has looked at whether this AI has the ability to improve the productivity of an employee in a company. Of course, checking, always the text generated by the AI, editing and correcting. Shakked Noy and Whitney Zhang of MIT (Massachusetts Institute of Technology) recently published the results of an empirical study on business professionals who used ChatGPT to write various business documents.

This was the chatGPT test

Study participants were 444 professionals with experience in various fields, including marketers, grant writers, data analysts, and human resource professionals. Each participant was assigned to write two business documents in their specialty: press releases, short reports and analysis plans, etc.

All participants first wrote a document in the normal way, without the help of the computer. At one point, half of the participants were randomly assigned to use ChatGPT to write a second document, while the other half wrote a second document as normal, without the help of AI.

Most of the participants were using the AI ​​tool for the first time. (30% of all participants had already used ChatGPT). “Normally, with any tool there is a learning curve: the more users use the tool, the more efficient they become at using it,” recalls the report.

Main conclusions

On the one hand, the study showed that ChatGPT is very easy to use for novice users, although the results are better among users with more experience with the tool.

After the business documents were drafted, their quality was rated on a scale of 1 to 7. Each document was rated by three independent experts.

One of the main conclusions is that “professionals who used ChatGPT were faster in preparing their documents and their quality was also higher“. In other words, it manages to tackle a common problem, which is a common conflict between working faster and getting good results (a phenomenon known in cognitive psychology as the trade-off between speed and accuracy).

The first round, in which the documents were produced without the help of AI, returned the same results in both groups. In other words, it was not the case that the participants in one group had more talent or ability than those in the other. In the second round of writing, large differences were observed, with better results in the group that used ChatGPT.

In the second round, professionals using ChatGPT produced their document in an average of 17 minutes, while professionals who wrote their document without AI help took 27 minutes. This represents an improvement in productivity of 59%.

“In other words, ChatGPT users could type 59% more documents in a workday than non-ChatGPT users, at least if all their writing involved only documents similar to those in this study,” say the researchers.

Regarding quality: the raters did not know which authors had received help from ChatGPT and the rated average quality of the papers, on a scale of 1 to 7, was much better when the authors had help from ChatGPT: 4.5 ( with AI) vs. 3.8 (without AI).

Less time writing, more time editing

Second, the practitioners were asked to report how they divided their time into three different phases of the writing process: brainstorming, writing a draft, and polishing this draft. Their responses suggested that the use of ChatGPT had changed the way users spent their time.

In this graph you can clearly see the distribution of time use:

chatGPT test

In the first round (without any help from the AI), professionals spent 25% of their time brainstorming, 50% writing a draft, and 25% editing to produce the final product. By using ChatGPT, participants possibly spent slightly less time brainstorming.

The time spent generating drafts was more than halved, as most of this workload was offloaded to ChatGPT. And interestingly, the time spent improving the draft doubled.

Leave a Reply