I realise I'm very late to the party, but I'm about to start making use of an LLM coding assistant, and I thought it would be interesting to capture my thoughts now, and then to compare after a couple of months.
My workplace is quite security-aware, so they've been careful about allowing developers to use LLM assistance. They've now starting to open up access to LLMs via AWS Bedrock, with Claude as a CLI tool. Some of my colleagues are already up & running, and I'll be getting it set up in the next week or so.
My experience of LLMs is almost non-existent. I haven't been permitted to use them at work, and I haven't felt any urge to use them at home. Aside from my ethical misgivings (more on that soon), I just haven't come across any ways of using them that have really appealed to me.
I'm sure there are other uses, but nothing's ever interested me enough to try. Ethically, I find the current market leaders hard to defend. In no particular order, here are some problems off the top of my head:
In short, the technology of the LLM is interesting and potentially useful, but the huge push for them at the moment is concerning, to say the least.
Given that I don't have much use for an LLM, and don't approve of the available services, I could happily stay well away for the forseeable future. Sadly, my current employer is expecting me to start using it. While I doubt they'll be tracking my usage and disciplining me for not using it, I'm sure I'd have to explain if I didn't.
Also, this job won't last for ever, and I'm sure interviewers will expect me to have experience with using LLM code assistants. They're out there, and as a professional, I'm obliged to keep my skills up to date. In this case, that means bringing myself up to a working level with LLMs.
As much as my colleagues rave about Claude, I don't expect it to transform my work. I've done some testing & code review for bugfixes from Claude on one of our apps, and they were sub-standard. The code for one looked OK, but didn't work. For the other, the approach was wrong, though it would still have been a marginal improvement over the current situation if it had worked. Again, it didn't.
I was also asked to review some Claude-generated documentation for the same app. Again, it looked OK at first glance, but was actually unhelpful - it was wrong, and would mislead anyone who tried to follow it.
So, based on those experiences and what I've read, I expect that a coding assistant will be useful for scaffolding and some boilerplate, should be helpful for the kind of common patterns I'd normally check via a search engine, and may do a better job of some refactoring than IntelliJ's standard tools. However, I'm expecting that the output will not be perfect, that I'll have to review changes carefully, and that I'll have to improve test coverage.
I don't expect it to revolutionise my working practices or to speed up my work significantly, and I don't expect to enjoy working with it.
I'll post a follow-up in a couple of months to report back on how the experience has been and whether it's met my expectations. I've set the bar pretty low, so maybe it'll exceed them and I'll be a born-again LLM evangelist!