GitHub Copilot, an AI-powered code completion tool, built on top of the OpenAI Codex, quickly rose to become a prominent name in today’s coding landscape. Copilot has captured substantial interest within the developer sphere for its capacity to boost coding productivity, offer insightful code recommendations, and simplify the overall development workflow. In our previous article, we provided a comprehensive guide to what it is and how to use it, preparing teams to start using it.

In our second follow-up article about GitHub Copilot, we will delve into the insights and experiences shared by software engineers and QA engineers who have utilized Copilot, an innovative coding assistant developed by OpenAI. To gain a deeper understanding of Copilot’s effectiveness and impact, we conducted a comprehensive survey, collecting valuable feedback from a diverse range of professionals. Continue reading to delve into the survey results and gain a comprehensive understanding of Copilot’s benefits, limitations, and the general sentiment of those who have adopted it.

50 Commencers (from our Development & QA teams) of different expertise and experience levels are asked to use Copilot for 2 months. To maximize Copilot’s benefits, all Software Engineers and QA Engineers were encouraged to volunteer and participate in its usage. It was planned to have a small group share their experiences with Copilot.

For those interested in using Copilot for the first time, a trial month was provided as part of an individual trial plan. Following the first month of use, all “first-time” users were required to provide structured insights to their managers.

In addition to voluntary usage, selected projects and platforms required all Software Engineers and QA Engineers to use Copilot and share insights for a longer assessment period. This assessment included all levels and roles, considering factors such as complexity and a new code base. The continuity of Copilot usage for these projects was evaluated based on the insights gathered.

Competency leaders were tasked with preparing guidelines to support and enhance their team’s Copilot usage. By preparing separate guidelines for development and QA, we documented on the Confluence Platform. The leaders were responsible for collecting feedback and guiding team members on effective usage through methods like team meetings and one-to-one sessions.

To provide organizational visibility, periodic overviews of Copilot usage were shared, ensuring transparency and awareness within the organization. Furthermore, to compare its performance with Copilot, a limited group tried Code Whisperer, enabling a comprehensive evaluation of its capabilities alongside Copilot.

With given use cases, participants share their insights through 2 surveys:

Copilot, while facing limitations in iOS development due to its dependency on Xcode, still maintains a strong user base. About %80 of all participants state that they would like to continue using Copilot. Its effectiveness shines in key use cases such as converting code to comments, converting comments to code, and writing functions from scratch.

Forecasts predict a 20% efficiency increase. Project complexity seems to have little impact on Copilot’s efficiency, particularly in auto-complete and handling simple code, although further assessment is required to validate this claim. In spite of the risk of taking shortcuts, it may actually be better to get their hands dirty as much as possible for less experienced developers and QA engineers.

The brush feature, aimed at improving code robustness, readability, and fixing simple bugs, has been inconsistent. Unfortunately, GitHub Copilot is not currently available in Android Studio. Additionally, 75% of participants expressed hesitation regarding whether Copilot surpasses ChatGPT, signaling a need for further evaluation and comparison between the two tools.

Copilot, trained with ChatGPT 4, has the potential to improve its performance. To see how this could enhance its capabilities, it would be worthwhile to explore this possibility. Currently, its usefulness is limited and often falls short of what is expected of it. Additionally, its context awareness is limited to active project files. However, there are instances where it saves time. One such case is when creating an entity from scratch, although it lacks efficiency when dealing with spring data attribute types and annotations. Once a method is written, Copilot learns and suggests similar algorithms, eliminating the need to search for the correct expression manually.

Copilot can generate test templates and offer code suggestions based on the context, reducing the time spent manually writing repetitive test code. This accelerates the test creation process and allows developers to focus more on crafting test cases specific to the functionality being tested.

The upcoming release of Copilot X holds promise for improvements that could significantly enhance the user experience. At this stage, the tool appears to be more beneficial for experienced engineers who may find it easier to identify the potential bugs introduced by Copilot. Junior programmers may find it challenging. However, the tool still offers a more convenient code-writing experience, especially when leveraging Copilot for repetitive tasks and quick line advancements using the “tab” key. It falls short of generating complete functions, requiring careful checking and verification.

Copilot for comments offers valuable assistance to developers. It serves as a starting point by suggesting code that can be modified and completed as needed. Copilot proves useful in reducing repetitive tasks, fixing simple logical errors, writing function descriptions, and swiftly completing similar code segments. Although developers haven’t extensively tested it with complex structures, demos have shown that even basic structures can present challenges. At times, Copilot provides accurate suggestions, while other times it may suggest irrelevant alternatives when attempting to complete written code.

The effectiveness of Copilot in handling complex methods and functions requires further evaluation. It appears to adapt to project structures and offers suggestions based on similar codes. In projects with intricate business logic, assessing their performance becomes more difficult. However, Copilot demonstrated usefulness and success in areas such as unit tests and converting comments to code. While developers haven’t had the opportunity to try refining code and converting code to comments, it is anticipated that Copilot will also be beneficial in those aspects. It generates readable code, particularly when working with utility functions.

While Copilot provides developers with different perspectives when writing code, it’s important to be cautious of incorrect suggestions. Furthermore, due to limitations in Xcode compatibility, developing code directly with Copilot is currently not possible. However, utilizing it in conjunction with VS Code for writing logic and unit tests proves helpful. Exploring the potential for Copilot to learn and implement base methods and tests would be an interesting avenue to pursue.

Overall, Copilot offers convenience and an enjoyable coding experience. Its potential for company-wide implementation is significant, in the belief that it will continue to develop and provide substantial benefits. Incorporating tests and test cases into its functionalities would further enhance its utility.

Here are a few comments regarding the usage of Copilot from the Commencis team:

While Copilot already offers assistance, the upcoming release of Copilot X is expected to provide significant improvements. With the introduction of Copilot X, developers can enhance productivity even further by leveraging its enhanced capabilities.

Copilot has proven to be an extremely beneficial tool, significantly improving coding efficiency and reducing task completion time. GPT-4 integration has elevated Copilot’s capabilities, making it a highly promising tool. However, for iOS projects, better integration with Xcode would improve efficiency.

Copilot excels at code completion, offering advanced suggestions beyond what the editor provides. However, its real strength lies in utilizing comments to instruct Copilot, rather than relying solely on code completion. It can be seen as an advanced version of VSCoder’s autocomplete, often suggesting example code. While this can be useful, it can also pose challenges in identifying brace errors within the suggested code.

The extent to which Copilot successfully scans the entire project remains uncertain. However, as I continued to use it, I noticed an improvement in the accuracy of its recommendations. Particularly in scenarios involving repetitive tasks, Copilot proves invaluable in boosting productivity. Recognizing patterns and implementing them throughout the entire flow would yield even greater efficiency gains.

Our Copilot usage survey sheds light on the experiences and perspectives of software engineers and QA engineers who have incorporated Copilot into their coding workflows. The feedback gathered provides valuable insights into the strengths and limitations of this innovative tool. Overall, Copilot has proven to be a beneficial coding assistant, offering improved efficiency and time savings for many users. The ability to generate code suggestions, convert comments to code, and help in writing functions has been particularly appreciated.

However, it is important to use Copilot at the right size and at the right point. Additionally, integration with specific development environments, such as Xcode for iOS development, remains a limitation that needs As Copilot continues to evolve and new versions are introduced, it holds great promise for further enhancing coding productivity and facilitating the development process. We hope that this survey and its findings provide valuable insights for development and QA teams as they explore Copilot’s potential in their coding endeavors.