Dartvel search: full text, hosted engines and semantic search

Search your models with typed results, from an in-memory index in tests to Meilisearch or OpenSearch in production.

Add semantic search when people search by meaning and the words do not match.

@DVModel(searchable: true) gives the model Article.search. It has no provider until you set one with Article.useSearchProvider, and a search before that throws a StateError.

Searches with: Your own database, with no search service to run. Use the one that matches your database engine

Article.useSearchProvider(MeilisearchProvider<Article, ArticleFacets>(
  baseUrl: Uri.parse('https://search.example.com'),
  apiKey: DV.Secrets.get('MEILISEARCH_KEY'),
  indexName: 'articles',
  fromJson: ArticleParser.fromJson,
  tuning: Article.searchTuning, // from dartvel.search in pubspec.yaml
));

Algolia has no local server, so its requests are checked against recorded ones.

final DVSearchResultPage<Article> page =
    await Article.search('dart', page: 1, perPage: 20);

for (int i = 0; i < page.items.length; i++) {
  final Article article = page.items[i];
  final String snippet = page.highlights.isEmpty ? '' : page.highlights[i];
  DV.log('${article.title}: $snippet');
}
DV.log('${page.total} matches, facets ${page.facetCounts}');

highlights has one entry per item, or none when the provider does not highlight.

# pubspec.yaml
dartvel:
  search:
    synonyms:
      refund: [return, chargeback]
    typoTolerance: true
    highlightPre: "<em>"
    highlightPost: "</em>"

Generation writes these into Article.searchTuning. Pass it to the provider so the settings live in one place. Synonyms work in both directions.

Mark fields with @DVModel.searchableField(). With none marked, every field is searchable.

A @DVModel.sensitiveField() is never indexed, even when it is marked searchable.

A tenant-scoped model searched through Postgres only returns the current tenant's records.

Semantic search finds records by meaning: "how do refunds work" finds an article about returning an order. Add semantic: true to @DVModel and the data model gets it.

// The model says it has one. The embedder and the vector store are the only
// things Dartvel cannot know, so they are the only things passed.
void semanticIndex() {
  Article.useSemanticSearch(
    embedder: DVAIEmbedder(
      OpenAIDVAIAdapter(apiKey: DV.Secrets.get('OPENAI_API_KEY')),
      id: 'openai/text-embedding-3-small',
      dimensions: 1536,
    ),
    vectors: DVInMemoryVectorAdapter(),
  );
}

// Saving queues the embedding. There is no second call, and nothing waits
// on the embedder: a worker drains the queue with
//
//     dartvel queue work --queue semantic
await article.save();

final DVSemanticPage<Article> page = await Article.semanticSearch(
  'how do refunds work',
  limit: 5,
);
for (final DVSemanticHit<Article> hit in page.hits) {
  DV.log('${hit.record.title} matched in ${hit.field}: ${hit.chunk?.text}');
}

The embedder and the vector store are the only two things Dartvel cannot know, so they are the only two you pass.

What is embedded is the prose the data model already declares: its searchable fields, its page title and its main content. A sensitive field is never embedded.

Saving a record queues an embedding job and destroying one removes it. Nothing is embedded during the save.

A worker does the embedding: dartvel queue work --queue semantic. Queries never wait on it.

mode is semantic (the default), keyword or hybrid. keyword and hybrid also read the search provider.

Long fields are split into chunks, and a record appears once with the chunk that matched.

You name the embedder and its dimensions. Vectors from two models cannot be compared, so changing the embedder builds a new index beside the old one.

DVInMemoryVectorAdapter is the only vector store. There is no pgvector or hosted vector adapter.

GETTING STARTED

APP

DATA

BACKEND

OPERATIONS

SHIPPING

REFERENCE

Search

Choose a search provider

Read results, highlights and facets

Tune synonyms, typos and highlights

Pick the fields that are indexed

Search by meaning with embeddings

Status