Tech Trends

Evaluating GitHub Copilot Chat Effectiveness: Insights from Akvelon

GitHub Copilot Chat Survey Results

How accurate is GitHub Copilot Chat? Can Chat compensate for the weaknesses of the Copilot in various areas? Does it increase performance? These were the questions Akvelon’s team was willing to delve into.

Our development teams often look for ways to speed up the processes while ensuring there’s no quality loss, so we are always ready to test how top-notch AI tools impact coding and testing. In this article, we’re willing to share the results of our survey.

GitHub Copilot Chat vs GitHub Copilot: What Is the Difference?

GitHub Copilot Chat enhances the coding experience by providing context-aware suggestions in a chat-like interface.

Unlike GitHub Copilot, which operates within an IDE, Copilot Chat offers a conversational format. Developers can engage in a dynamic dialogue, seeking suggestions, clarifications, and alternatives. This more human-like interaction expands the capabilities of GitHub Copilot and fosters an intuitive collaboration with an AI-powered coding assistant.

Both tools aim to streamline coding and reduce cognitive load, but GitHub Copilot Chat adds interactivity for a more personalized and conversational experience. Developers can ask Copilot Chat to write code, explain code, and even learn from it. Understanding their distinctions empowers developers to make informed decisions about integrating them into their workflow. That is why the Akvelon team has embraced an opportunity to test them both out. Keep reading to learn about GitHub Copilot Chat performance.

Akvelon’s Survey Participants and Their Roles

In our survey on GitHub Copilot Chat effectiveness, we gathered responses from a diverse group of participants representing various roles within the software development field. The participants included full-stack developers, front-end developers, data engineers, DevOps specialists, and back-end developers. Their collective expertise and experience provided valuable insights into the usage and perceptions of the GitHub Copilot Chat feature across different domains of software development.

GitHub Copilot Chat Survey: Developers Roles

The survey included participants with varying levels of experience from junior (1+ years),  to senior (8-12+ years). Their feedback highlighted Copilot Chat's potential for enhancing productivity across different experience levels.

The survey on GitHub Copilot Chat effectiveness included participants who primarily utilize a diverse range of programming languages, such as C#, JavaScript, TypeScript, Python, Shell Scripting, Terraform, Azure Bicep, and Java. The insights shared by participants demonstrate how GitHub Copilot Chat effectively supports developers working with different languages, offering assistance and enhancing productivity in their respective technology ecosystems.

GitHub Copilot Chat Survey: Used Programming Languages

GitHub Copilot Chat Features and Effectiveness [Akvelon’s Experience]

The survey participants were asked about the areas in which they believe GitHub Copilot Chat could provide the most value to them. Their responses showcased a variety of perspectives and highlighted the diverse range of functionalities that developers find valuable.

Code generation

GitHub Copilot Chat enables developers to interact conversationally with the AI assistant. It allows developers to request code, explain code, and learn from it.

Code generation has a positive impact on participants' code quality and their ability to generate code efficiently. Most developers found the code suggestions to be accurate and helpful, while others utilized them for specific tasks like generating sample logic or writing unit tests.

Most developers have reported a remarkable acceleration in the code generation of 2-3 times. They could generate code snippets and suggestions seamlessly, saving valuable time and effort.

GitHub Copilot Chat: Code Generation

Code explanation

GitHub Copilot Chat provides one more useful feature – the ability to request explanations for specific code segments. Code explanation serves as an invaluable learning tool, particularly for beginners and when tackling unfamiliar languages or frameworks.

However, developers did not utilize or find it helpful for regular code parts. Overall, GitHub Copilot Chat effectiveness depends on the context and complexity of the code, with different participants reporting increases ranging from 11-25% to more than 75%.

GitHub Copilot Chat Survey: Code Explanation

Language translation

GitHub Copilot Chat is able to translate code from one programming language to another. This feature is often helpful when developers come across a code snippet solution in a different language and need to seamlessly translate it to match the target language of the repository.

Most respondents did not use this feature or did not find sufficient use cases to test it out. As a result, they did not experience any notable impact on their ability to convert code from one language to another.

However, one participant mentioned utilizing the language translation feature to transform code from Terraform to Azure Bicep. He reported a moderate increase of 26-50% in their ability to convert code between these specific languages.

GitHub Copilot Chat Survey: Language Translation

Debugging

GitHub Copilot Chat's error detection and correction system uses machine learning algorithms to identify and fix errors in code. It handles syntax, logic, and coding style issues, supporting multiple officially supported programming languages.

Some of Akvelon’s developers found debugging helpful, experiencing increased speed in identifying and fixing errors by 25%. They utilized the feature for debugging complex code and received suggestions for adding logs. However, others found limited use or no specific increase in their debugging speed.

Refactoring

GitHub Copilot Chat excels in code refactoring and maintenance due to its contextual understanding of your code. Trained on vast amounts of public repository code, it possesses a comprehensive knowledge of programming languages, frameworks, and libraries.

The Refactoring feature in GitHub Copilot Chat had a diverse impact. 37.5% of participants found it helpful, stating an increase of up to 10% in code quality and speed of refactoring.

In many cases, it suggests more concise solutions, that are better. But in some cases, for negligible performance gains, it can complicate code a lot.

GitHub Copilot Chat Survey: Refactoring

Test generation

The impact of the Test generation feature varied among users. 12.5% of developers experienced an 11-25% increase in unit test creation speed for their database and FastAPI app.

However, one user found the generated tests to be unsatisfactory, with incorrect syntax and limited coverage. Another user with a shared testing concept in their codebase saw a significant increase of over 75% in test creation speed. Overall, the feature had mixed outcomes, with varying impacts on code quality and test creation speed.

Code review

GitHub Copilot Chat offers code review and suggestion features that help identify and address potential issues in the code.

37.5% of Akvelon’s survey participants experienced an 11-25% increase in code review speed. However, while GitHub Copilot Chat provided helpful recommendations, the generated code sometimes required significant corrections and resulted in lower code coverage. Nonetheless, it instilled confidence in double-checking the code before it’s published.

GitHub Copilot Chat Survey: Code Review

How GitHub Copilot Chat Accelerates the Development Process

62.5% of Akvelon’s survey respondents find the Refactoring feature the most useful. 50% of them also mentioned the Code generation feature.

The "chat" feature is nice, although I didn't utilize it as frequently. However, the regular code suggestions provided by Copilot are FIRE! Sometimes, it's astonishing how accurately it can anticipate your thoughts, and that really saves time.

The survey findings indicate that both Refactoring and Code generation features are highly regarded and valued by the respondents. However, the Refactoring feature stands out as the most preferred option, showing a significant preference over Code generation among the developers.

GitHub Copilot Chat Survey: Most Valuable Feature

The time-saving potential of GitHub Copilot Chat varies based on such factors as task complexity, developer proficiency, and familiarity with the programming language.

Most of our team reported that it’s possible to save only about 1-3 hours per week, depending on individual experiences.

GitHub Copilot Chat Survey: Saved Hours

87.5% of developers in Akvelon claimed that GitHub Copilot Chat is a valuable tool that can enhance productivity and accelerate the coding process. Numerous developers have benefited from its ability to reduce the time spent on repetitive or boilerplate code, making it a helpful asset in software development.

GitHub Copilot Chat Survey: Recommended?

Ultimately, the decision to use GitHub Copilot Chat depends on personal preference and workflow. Some developers may find it beneficial and time-saving, while others may prefer to rely on their own coding skills.

87% of developers claimed that GitHub Copilot effectively reduced their mental effort when tackling repetitive tasks. This highlights the beneficial impact of AI coding tools, as they enable developers to conserve cognitive energy and concentrate on the more demanding and inventive aspects of software development.

Separately, GitHub Copilot and GitHub Copilot Chat enhance development speed and quality, but their combined usage produces an even more significant impact. For instance, Copilot Chat excels at tasks such as refactoring, code review, and code translation across different programming languages. On the other hand, GitHub Copilot was not specifically designed to handle these functionalities.

The Bottom Line

Akvelon's survey revealed that many developers find GitHub Copilot Chat useful and experience its benefits, with potential time savings ranging from less than 1 hour to 11-20 hours per week.

However, validating and reviewing the code generated by GitHub Copilot Chat is still very crucial for developers, so they have to ensure its accuracy and alignment with project requirements. At Akvelon, we have established procedures to prioritize code security and seek client consent for the utilization of AI tools in their projects. We conduct meticulous analysis of our project to determine the optimal use of AI tools in software development.

When partnering with Akvelon, you can expect the expertise of a pioneering service provider who understands how to safely leverage the benefits of AI in your projects. Let us assist you in delivering features more efficiently by utilizing our proficiency with GitHub Copilot.

This article was written by:
Ilya

Ilya Polishchuk

Director of Engineering