GitHub CoPilot works well for various languages, libraries, and frameworks, including Python’s pandas, NumPy, Django, and FastAPI. Adroiti colleague Michael Nelson uses it every day when writing code, both for data science research as well as for software development. According to Michael, the main advantage of CoPilot is its powerful code completion, but there are also risks and other challenges to keep in mind.

The main challenge – reliability

When talking about the complexity of working with GitHub CoPilot, Michael says that it is pretty simple to use GitHub CoPilot – it has integrations with most major IDEs. The challenge for everyone, but particularly for junior-level professionals, is that the AI’s suggestions aren’t guaranteed to be correct, especially when it comes to implementing business-specific requirements. Thus, careful evaluation of each AI suggestion is required, which is often easier for those with more years of experience.

“Everyone on my team is using CoPilot effectively, from juniors to seniors, though you may take that with a grain of salt since the juniors on my teams are quite talented!

The key to using any coding AI assistant is to specify the desired behaviour of the code. One effective way to accomplish this is by writing clear comments and/or docstrings, as well as choosing informative variable names.

Of course, this is always a best practice anyway, but with a coding AI assistant, it’s doubly effective since the code may then end up writing itself! As far as results, we haven’t gone to the extent of rigorously measuring the productivity impact of GitHub CoPilot. However, we have heard people report that the AI CoPilot is generating 40% or more of their lines of code”, – says Michael.

Tips for Defining Desired Behavior

When it comes to defining desired behaviour, a key concept that has gained significant attention is ‘Prompt engineering’. This approach, which has been the focus of numerous articles and even full courses, is about effectively specifying the desired behaviour. Michael shares a few tips from Microsoft’s CoPilot tutorial, which they refer to as the 4S plus a fifth S, a concept he himself has added:

• Make each prompt (or comment) concise, specific, and focused on a single task. This helps to avoid ambiguity and ensures the AI's understanding of the desired behaviour.
• Surround the new code with relevant information, including keeping related files open in the editor, because open files will also be analyzed by the AI when generating suggestions.
• Show examples, e.g. when defining a function, give an example of the input and expected output.

“As far as results, we haven’t gone to the extent of rigorously measuring the productivity impact of GitHub CoPilot. However, we have heard people report that the AI CoPilot is generating 40% or more of their lines of code”, – says Michael.

The Value of CoPilot

The core utility of GitHub CoPilot is its powerful code completion.

“Often, I can finish an entire line of code, if not multiple lines of code, with a single keystroke. This is great for efficiency, as well as for staying focused on the bigger picture of whatever problem I’m trying to solve. The reduction of keystrokes is also beneficial for those concerned with repetitive strain injury”, – says Michael, highlighting several other benefits of working with language-based AI like CoPilot:

• Saving time, energy, and focus with code completion
• Generating additional ideas. It can be beneficial when creating test cases.
• It helps if you are "stuck" for whatever reason – be it troubleshooting a particularly sneaky bug, seeking a more elegant solution to a problem, or working with a new (unfamiliar) software package. The AI assistant's ability to provide ideas that may not have come to mind has been invaluable."
• Articulating the problem to the AI assistant – it often clarifies it within my mind. In this respect, the AI assistant is an excellent upgrade to rubber ducking (where a developer explains their code to a rubber duck) – unlike the duck, the AI can talk back!"

„AI solutions, in general, have expedited research and development processes at Adroiti, enhancing quality and fostering unique innovations. We have used many forms of AI. In addition to GitHub CoPilot, other new large language models like ChatGPT and GPT4 have proven great for research and development. For example, it can generate suggestions of software libraries to solve specific problems, along with examples of how to use those libraries. While there may often be errors or outdated information in the responses, it still almost always gives some reasonable ideas that serve as a good starting point.

We have, of course, also leveraged other kinds of AI models for solving specific business problems, such as extracting text from images, forecasting price changes, and predicting demand for products. Other teams at Adroiti have done fantastic work with handwritten text, for example, determining the author of historical texts based on their handwriting. There are lots of other cool things in the works, but as you said, we can't disclose the details ", – Michael talks about the practical use of these tools.

Key Risks and Challenges

Michael says that it's very easy to get comfortable with the AI's suggestions, which often seem just right. But this comfort can lead to complacency, where you might miss errors or unintended consequences hidden in the code. Moreover, CoPilot, by its nature, will repeat patterns it has seen, and this can include repeating bugs that were written by humans, as well as other undesirable things like violating coding conventions.

Another risk to be mindful of is the security of secrets and proprietary information.

"At the core of the problem is that most AI assistants, including GitHub CoPilot, run in the cloud, so your code must be sent to the cloud. Recall that this might include any files open in your IDE, even files with secrets. GitHub CoPilot offers ways to mitigate this risk, but it must always be considered when dealing with sensitive or proprietary information. One particularly useful feature for mitigating this risk is a configurable whitelist for which types of files the CoPilot is allowed to analyze ", - says Michael.

Essential Skills for Success

„Things are developing so rapidly that it’s hard to say exactly where everything will end up. All experts agree that AI will become much faster at completing specific tasks. The hardware and software running the models will get faster. In some cases, they will even run directly on individual computers and phones, removing network lag and the need for an internet connection. This means that tasks that might be impractically slow now could be pretty fast in the future. As such, it’s worth investing time to learn what an AI can do for you, even if it feels a bit too slow to be helpful right now, – says Michael, talking about the future of the AI revolution.

Among Michael, AI will also inevitably gain access to additional relevant information about the specific task or project being worked on. This should lead to increasingly accurate and relevant AI responses. Individuals and organizations can get the most out of this capability by having their organization’s knowledge and data clearly and thoroughly documented.

A few essential skills to succeed with AI tools are:

• Technical communication. It is the essence of interactions with new AI tools; the more concisely – briefly but comprehensively – one can communicate the desired result, the better and faster the result from the AI should be.
• Critical thinking. It is essential to formulate the desired result and rigorously evaluate the result generated by the AI.
• Adaptability. It is perhaps the most critical since AI presents many new approaches to how work is done.”

“IT professionals have a lot of ways to enhance these skills in their current day-to-day work. Those who invest time in reviewing their colleagues’ code will have an edge in reviewing AI-generated code. Putting extra effort into clear and precise documentation – whether for code, research results, or product specifications – will pay dividends when the time comes to communicate information to and generate results from an internet of AIs.”

Follow our latest insights on Adroiti Linkedin!