juanrivillas.com

Recap

If you're just arriving here, let me get you up to speed in a few sentences. I'm working on a project to render a family visualization tree, and I'm implementing an AI system that responds to questions about my family. In the first implementation, I tried a naive RAG system with embeddings. In this post, I discuss my experience while building that feature. Then, I modified the original implementation to use Graph RAG. In this post, I compare both implementations. In today's post, I will discuss how I enhanced the responses from the Graph RAG implementation through prompt engineering techniques.

Prompt fine-tuning

Prompt fine-tuning is the process of different prompt engineering techniques that enrich the context of your LLM call with information relevant to the problem you're trying to solve.

In my case, I used prompt fine-tuning in every tool to instruct my model on the purpose of each tool. This technique is an underrated feature because it plays a crucial role in the process of solving problems with LLMs. The context and instructions you provide are paramount for a successful prediction.

In my implementation, I used two tools:

One to generate a Cypher query
Another one to execute that Cypher query into Neo4J

The latter was trivial. The former, though, was a bit more complex. Since I was using a general-purpose chat model, such as gemma3n:latest, the outcome of a call attempting to generate a query was somewhat unpredictable. It could work, but it could not work too. Oftentimes, it didn't by the way. Therefore, I had two options

Fine-tune my prompt
Convert the existing cypher_generator_tool into an agent, and create multiple tools for every access pattern

I wanted to learn and explore the first option further, so that's what I chose. In another situation, such as a production-ready environment, I would have gone for the second option, as it creates a more predictable outcome. However, I wanted to see how far I could get with prompt fine-tuning.

The process of fine-tuning

I planned my problem as follows.

I created a list of access patterns, similar to what we do in real applications with REST endpoints and controllers. If you're not familiar with the concept of an access pattern, I'm referring to considering how users will access your data. I anticipated questions like
1. What's the relationship between person A and person B
2. Who's person A grandfather/grandmother
3. Who are person A's siblings
4. and more..

The questions that can be answered directly from the graph labels are trivial so that I won't explore them right now. Instead, I wanted to focus on indirect relationships that had to be inferred through multiple calls or more complex queries, such as let's find the shortest path between person A and person B, which are derived from graph theory and supported by Neo4J.

The first step was to instruct my LLM about the DB schema. For the family-tree problem, that part was easy because the schema was small and concise.

Node Types:
- Person: Represents a family member
  Properties: name (string), birth_date (date), death_date (date), bio (string), occupation (string), location (string)

Relationship Types (Keep it Simple):
- PARENT_OF: Connects a parent to their child (directional: parent -> child)
- MARRIED_TO: Connects spouses (bidirectional)

Next, I informed some of the known access patterns.

Smart Query Patterns (Derive Complex Relationships from Basic Ones):

1. Find a person by name (exact match):
    MATCH (p:Person {name: "Juan Pablo Rivillas Ospina"}) RETURN p

2. Find a person by name (partial match):
    MATCH (p:Person) WHERE p.name CONTAINS "Juan Pablo" RETURN p

3. Find all children of a person:
    MATCH (parent:Person {name: "Juan Pablo Rivillas Ospina"})-[:PARENT_OF]->(child:Person) RETURN child

4. Find parents of a person:
    MATCH (parent:Person)-[:PARENT_OF]->(child:Person {name: "Joao Rivillas de Magalhaes"}) RETURN parent

5. Find spouse of a person:
    MATCH (p1:Person {name: "Juan Pablo Rivillas Ospina"})-[:MARRIED_TO]-(p2:Person) RETURN p2

6. Find siblings (children of same parents):
    MATCH (person:Person {name: "Joao Rivillas de Magalhaes"})<-[:PARENT_OF]-(parent:Person)
    MATCH (parent)-[:PARENT_OF]->(sibling:Person)
    WHERE sibling <> person
    RETURN DISTINCT sibling
...

See how I also pointed the LLM to infer the more complex access patterns from the simple ones. What we're building here is much more than context. It's a tool set for the LLM to use in its favor!

Finally, I provided a list of what I considered advanced patterns.

Advanced Patterns:
- Use variable-length paths [:PARENT_OF*1..10] for recursive queries
- Use OPTIONAL MATCH for relationships that might not exist
- Use UNION to combine multiple relationship patterns
- Use collect() and DISTINCT to group related results
- Use length() to calculate relationship distance
- Use ORDER BY to sort by generation distance

This approach suits my problem because the scope is well-defined.

Before vs After: Real Examples

Let me show you the concrete impact of prompt engineering with actual examples from my implementation.

Example 1: Finding Relationships Between People

User Question: "What is the relationship between Juan Pablo Rivillas Ospina and David Rivillas de Magalhaes?"

Before Prompt Engineering:

MATCH (a:Person {name: 'Juan Pablo Rivillas Ospina'})-[r:RELATIONSHIP]->(b:Person {name: 'David Rivillas de Magalhaes'})

Problem: This query assumes a generic 'RELATIONSHIP' type that doesn't exist in our schema (we only have PARENT_OF and MARRIED_TO), so it returns no results. The LLM invented a relationship type instead of using the actual schema.

After Prompt Engineering:

MATCH path = shortestPath((p1:Person {name: "Juan Pablo Rivillas Ospina"})-[*1..6]- (p2:Person {name: "David Rivillas de Magalhaes"}))
RETURN path, [r in relationships(path) | type(r)] as relationship_types, length(path) as path_length

Result: Successfully identifies the relationship path and shows Juan as David's father with the connecting relationship.

Example 2: Finding Siblings

User Question: "Who are David Rivillas de Magalhaes siblings?"

Before Prompt Engineering:

MATCH (p:Person {name: "David Rivillas de Magalhaes"})-[:PARENT_OF*1..1]->(parent)
WITH parent
MATCH (parent)-[:PARENT_OF*1..1]->(sibling)
WITH sibling
MATCH (sibling)-[:PARENT_OF*1..1]->(grandparent)
WITH sibling, grandparent
MATCH (p)-[:PARENT_OF*1..1]->(grandparent)
WITH p, grandparent, sibling
RETURN DISTINCT p.name AS sibling_name, p.birth_date AS birth_date, p.bio AS bio, p.occupation AS occupation, p.location AS location
LIMIT 20

Problem: It generates an overly complex query, even looking at the grandparent, which is not what we wanted.

After Prompt Engineering:

MATCH (person:Person {name: "David Rivillas de Magalhaes"})<-[:PARENT_OF]-(parent:Person)-[:PARENT_OF]->(sibling:Person)
WHERE sibling <> person
RETURN DISTINCT sibling.name

Result: Correctly finds all of David's siblings by identifying shared parents.

Results

A few insights from this process were:

Letting the LLM decide what query to generate based on the schema was enough for the obvious queries:

Who's person A's spouse?
Who is person B?
What are the hobbies of person C?

However, for complex relationships, it would likely fail. When I say 'fail,' I mean that the query would run, but the outcome was not what I desired. The LLM indeed generated a syntactically valid query but not a semantically correct one.

As a result, this process would fail most of the time for complex queries.

Some access patterns required special wording, such as

Find relationship path between two people (CRITICAL - use this format EXACTLY):

I wanted the LLM to use the query I provided precisely.

Although I decided not to use proper tool calls for this step, it is something I would definitely use in a production environment because it also allows one to extend the architecture more easily. Let's recall that the context has a character limit that depends on the model. So use this technique wisely.
Using prompt fine-tuning is not mutually exclusive with tools calls. In fact, real production systems use both.

Conclusion

Prompt engineering proved to be a powerful technique for improving Graph RAG performance in my family tree project. By providing structured schema information, access patterns, and advanced query examples, I was able to significantly enhance the LLM's ability to generate semantically correct Cypher queries.

Key Takeaways

When Prompt Engineering Works Well:

Well-defined, limited domain (like family relationships)
Clear schema with predictable patterns
Moderate complexity queries that benefit from examples

When to Consider Alternatives:

Complex domains with many edge cases
Systems that need to scale beyond context window limits

Best Practices I Learned:

Start with schema: Always provide complete, accurate schema information
Show, don't just tell: Include concrete query examples for complex patterns
Use critical annotations: Mark essential patterns with "CRITICAL - use this format EXACTLY"
Plan for failure: Have fallback strategies when prompt engineering isn't enough

Looking Forward

While prompt engineering significantly improved my Graph RAG system, it's not a silver bullet. For production systems, I'd recommend a hybrid approach: use prompt engineering for flexibility and tool-based agents for critical, high-reliability operations.

In my next exploration, I plan to convert this system into a proper agent architecture with specialized tools for different query patterns, combining the best of both approaches.

Interested in the implementation? Check out the full code on GitHub.