Tech Trends

GitHub Copilot Efficiency Explored: Key Takeaways from Akvelon’s Survey

GitHub Copilot Efficiency

Today’s fast-paced business environment makes CIOs and IT leaders constantly boost time to value and speed up software release cycles. As mentioned in Gartner’s press release, businesses are expected to deliver digital dividends as never before using every possible digital channel and asset. Outsourcing qualified engineers is just half the battle to optimize development costs. Finding teams that master AI-powered tools to increase their productivity is equally important. Akvelon’s engineers have recently spent over 8 000 hours investigating GitHub Copilot’s efficiency under real-life conditions and come to valuable data-driven conclusions.

The GitHub Copilot team has stated that their tool boosts productivity and reduces time for routine tasks. Akvelon couldn’t stay aside from exploring how true these statements are for engineers’ regular workflow. So we’ve launched a massive investigation project where 30 experienced developers have been using GitHub Copilot for their various day-to-day activities.

Further come the survey results from our GitHub Copilot application and quite unique insights on how the use of the tool affects the delivery time of real-life projects.

What Can GitHub Copilot Do To Accelerate Project Development

GitHub Copilot, like a proper second pilot, picks up on the context of the code, and instantly suggests corresponding code lines and completes functions. It is trained to adapt to the libraries in use, giving developers an advantage in their day-to-day routine. At least these are the benefits stated by the GitHub Copilot creators.

GitHub Copilot survey participants

We’ve decided to field-test GitHub Copilot and see how much value it can bring through the actual situations software development teams face. 30 Akvelon engineers have been using GitHub Copilot daily within their working routine. Most survey members have tested the tool for around a month and up to half a year on several languages and tasks of different complexities.
For how long the survey participants used github copilot
About 38% of the participants had up to 3 years of programming experience when they took part in the survey. The rest of the engineers had been working in the field for more than 4 years, some reaching the mark of over 8 years of experience.
The seniority of the GitHub survey participants
Most of the developers were handling back-end related tasks on the projects where they had a chance to test GitHub Copilot. A smaller number of participants were involved with full-stack development. Roughly 10% of participants worked on the front-end part, and the other 10% – on data engineering.
GitHub Copilot users statistics
With a better understanding of our survey participants' experience levels and key areas of involvement in projects, we get closer to uncovering the outcomes. Now, let's explore which programming languages were used to test GitHub Copilot efficiency.

Programming languages used for testing GitHub Copilot

We had a chance to explore GitHub Copilot efficiency, applying it to only some programming languages among many due to the projects the participants were working on during the survey. Most developers (roughly 59%) have been using GitHub Copilot to handle back-end tasks involving programming on Python, C#, and Java. Around 30% battle-tested the tool programming on JavaScript and TypeScript.
GitHub Copilot programming languages
We do not doubt that GitHub Copilot has the potential for successful application with other popular programming languages like PHP, Go, C++, Ruby, etc. However, at this point, the contributors to our survey had no opportunity to field-test GitHub Copilot for those.

Unlocking GitHub Copilot’s Power: Whom It Benefits the Most

The technologies and specifics of the projects that Akvelon worked on at the time of the survey dictated the conditions for our experiment. Also, the seniority level, the skillset, and the goals of the participants varied, which influenced their success in using the tool.

Despite the fact we had some limitations, for example, in programming languages applied, we’ve challenged GitHub Copilot in a way. We tested the tool for front-end and back-end programming and even data processing.

How do you use github copilot

Our observations revealed that GitHub Copilot brought the most value to those struggling with routine tasks and repetitive code patterns for both: front-end and back-end parts. Developers got proper code pieces suggested automatically based on what they've already written. GitHub Copilot also eased coding, completing methods and functions for developers. Finally, the process of creating unit tests got simplified to a noticeable extent.

When survey participants face more complex tasks like writing algorithms, it seems GitHub still lacks training for this kind of work. Sometimes the tool makes false suggestions which takes extra time for an engineer to validate the provided code.

GitHub Copilot may not fully meet all user expectations at present. However, the tool is continuously advancing and progressing as it faces more challenges, resulting in increased efficiency.

How GitHub Copilot Boosts Productivity: Akvelon’s Results vs GitHub Stats

GitHub Copilot promises to bring the spark to the development process that engineers miss due to routine boilerplate code writing. As mentioned earlier, the tool helps streamline typical code writing and increases productivity completing functions for the developers. Within our GitHub Copilot survey, we’ve been checking the tool for code writing quality and productivity boost.

Let’s first see what survey results say about the relevancy of GitHub Copilot’s suggestions.

How often github copilot suggestions are accurate

According to most survey participants, the suggestions provided by GitHub Copilot were quite accurate. Developers were able to incorporate these suggestions into their code right away. However, for more unique or specialized tasks, additional manual validation was needed to ensure accuracy.

Next, we received feedback from the survey respondents regarding the increase in coding speed. This aspect has the potential to impact the overall productivity of developers.

Coding speed increase with GitHub Copilot

Most respondents reported a coding speed improvement of up to 25%, while nearly 14% experienced an even greater increase of up to 50%.

The majority of the survey participants (81%) have witnessed that their productivity grew while they were using GitHub Copilot for coding and testing. The average time reduction rate for completing tasks with GitHub Copilot was 16% according to our survey results. Yet some developers have experienced a boost of up to 50%.

Increase of productivity from using github copilot
Despite the fact that GitHub Copilot fairly solves just part of the developers’ frustrations and has much space for improvement, it reduces the overall tension associated with repetitive tasks. Engineers can cut the routine and save more energy for building complex unique solutions. The tool also gives a helping hand in simple things like language syntax that may just slip out of developers’ mind if they work with several languages and projects. The feeling that you’re backed up just in case is hard to overestimate.

Survey Update: More GitHub Copilot Efficiency Zones Unlocked

After conducting our initial GitHub Copilot efficiency survey, we further investigated how this tool can enhance productivity across different roles within software development teams. The findings demonstrated GitHub Copilot's effectiveness beyond just code writing and unit test creation, enabling us to expand the survey with more insights.

Unit test writing with GitHub Copilot
Code refactoring with GitHub Copilot
Onboarding to new languages and libraries with GitHub Copilot
DevOps with GitHub Copilot
Documentation writing with GitHub Copilot

The tool significantly speeds up code refactoring and maintenance, aids in script development and automation within DevOps workflows, facilitates project documentation creation, and assists developers in getting up to speed with new libraries and languages during onboarding.

What Are the Possible Concerns Associated with Using GitHub Copilot?

GitHub Copilot is still quite fresh on the market, and its ability to provide value hugely varies on the engineers’ experience, language in use, and project specifics. The tool significantly reduces the need for manual effort for typical programmer tasks, streamlining coding and bug fixing. But there are particular cases when GitHub Copilot can rather interfere with the job to be done smoothly than help.

Let’s take a closer look at the key concerns associated with using GitHub Copilot and how we address them at Akvelon.

GitHub Copilot may give false code suggestions that mislead developers

Let's be real, among early GitHub Copilot adopters, there have been instances where the suggested code options may not align perfectly with the purpose of the solution. For example, GitHub Copilot may not efficiently assist in utilizing newly-arrived libraries and frameworks. The tool was trained on a predefined set of source code patterns, and it requires further training to pick up on the latest libraries and stay up to date.

Based on the findings from the Akvelon survey, one-third of participants reported that GitHub Copilot's suggestions had up to 80% of accuracy. But in case developers know exactly what solution they’re going for, even 50% of accuracy from the tool may be helpful. Then developers browse through all given recommendations and pick up the optimal one. According to our survey, engineers with more than a year of experience can identify issues or errors in the code, choosing the best way to go for their domain and context.

The key message here is not to treat GitHub Copilot as a miracle worker who delivers perfectly sculptured end results, but rather pick the most suitable perks out of many options. It’s still on developers to leverage their expertise and judgment to avoid misleading suggestions and detect truly valuable input.

The same principle applies when using GitHub Copilot to enhance code quality. According to our survey, a majority of participants noticed a slightly positive impact on their code.

Approximately 7% noted a slightly negative effect which might be related to the fact that they haven’t had enough time to master the tool. It’s important to understand that the negative figure can diminish as engineers gain more practice with GitHub Copilot and their overall professional experience grows.

What does GitHub Copilot do for code quality
Developers with firm knowledge and experience in their domain field can leverage GitHub Copilot to its fullest potential. Their expertise allows them to disregard unsuitable suggestions, ensuring optimal tool usage. Therefore, before using GitHub Copilot on live projects, every team should give it a thought and try to make sure they reached a stage where adopting the tool can benefit their workflow.

Security vulnerabilities in the code suggestions

Just like GitHub Copilot users may face trouble with non-optimal code suggestions, they may run upon code that contains security vulnerabilities. As previously mentioned, it is important to review the suggested code before using it, the same applies to security matters. We insist that only experienced programmers capable of addressing vulnerability concerns should rely on GitHub Copilot code suggestions. However, it's worth noting that security issues are not exclusive to AI-generated code.

Additionally, GitHub Copilot continually updates its algorithms with the recent launch of an AI-driven vulnerability prevention system that actively detects and blocks insecure coding patterns in real time. This ongoing effort helps to mitigate security risks associated with the suggestions provided by GitHub Copilot.

GitHub Copilot raising potential legal issues for business

GitHub Copilot was trained on open-source repositories. Many users felt concerned that the tool would provide code suggestions that violate copyrights putting companies at risk if they integrate someone’s code into their production software. However, GitHub Copilot does not extract code from repositories to present it as its own suggestions.

Furthermore, GitHub Copilot users can disable suggestions that match the code found in public repositories, mitigating any copyright-related risks.

Just a reminder, you shouldn’t expect from GitHub Copilot to generate complex algorithms independently. It rather helps to complete common functions and methods, reducing repetitive tasks.

Why Should CIOs Seek Teams That Embrace GitHub Copilot

Leveraging cutting-edge technologies and approaches is crucial for businesses to stay competitive. Teams with a certain level of expertise with AI tools already explore GitHub Copilot efficiency and reach higher productivity while delivering better customer experiences as a result. At this point, even a 10% increase in productivity can impact the speed of deployment in the long run, which only deepens our passion for cutting-edge tools!

By partnering with Akvelon, you’ll find a pioneering services provider who already knows how to use benefits from AI safely in your projects. We’ll help you deliver features faster using our expertise with GitHub Copilot.

This article was written by:


Ilya Polishchuk

Director of Engineering