☆ Yσɠƚԋσʂ ☆

  • 16K Posts
  • 12.8K Comments
Joined 6 years ago
Cake day: January 18th, 2020
  • ☆ Yσɠƚԋσʂ ☆@lemmy.mltoAI@lemmy.ml•We Asked 100+ AI Models to Write Code. The Results: AI-generated Code That Works, But Isn’t Safe
    2·
    4 hours ago

    Using LLMs does not obviate the need for the human user to understand what the code is doing. I’ve also found that, as with any tool, it takes time to actually learn to use LLMs effectively.

    In particular, I find it’s really important to understand the problem being solved, and then come up with the solution yourself. One approach I’ve found to be effective is to stub out the functions myself, and have the agent fill in the blanks for me. This helps focus the LLM and prevent it from going off into the weeds.

    Another trick I find handy is to ask the agent to first write a plan for the solution. Then I can review the plan and tell the agent to adjust it as needed before implementing. Agents are also pretty good at writing tests, and tests are much easier to evaluate for correctness because good tests are just independent functions that do one thing and don’t have a deep call stack. My current approach is to get the LLM to write the plan, add tests, and then focus on making sure I understand the tests and that they pass. At that point I have a fairly high degree of confidence that the code is indeed doing what’s needed. The tests act as a contract for the agent to fill and as a specification for the defined functionality.

    I suspect that programming languages might start shifting in the direction of contracts in general. I can see stuff like this becoming the norm, where you specify the signature for the function, and you could also specify parameters like computational complexity and memory usage. The agent could then try to figure out how to fill the contract you’ve defined. It would be akin to genetic algorithm approach where the agent could converge on a solution over time. If that’s the direction things will be moving in, then current skills could be akin to being able to write assembly by hand. Useful in some niche situations, but not necessary vast majority of the time.

    Finally, it’s very helpful to structure things using small components components that can be tested independently and composed together to build bigger things. As long as the component functions in the intended way, I don’t necessarily care about the quality of the code internally. I can treat them as black boxes as long as they’re doing what’s expected at the surface level. This is already the approach we take with libraries as we don’t audit every line of code in a library we include in a project. We just look at its surface level API provided.

    Incidentally, I’m noticing that functional style seems to work really well here. Having an assembly line of pure functions naturally breaks up a problem into small building blocks that you can reason about in isolation. It’s kind of like putting Lego blocks together. The advantage over stuff like microservies here is that you don’t have to deal with the complexity of orchestration and communication between the services.

Moderates