The first version of any search feature is a LIKE:

SELECT id, title FROM substances WHERE name LIKE '%' + @q + '%';

It works for about a week. Then somebody searches for aspirin in a corpus that spells it acetylsalicylic acid, somebody else pastes in C9H8O4 and gets nothing, a third person types 2-(acetyloxy)benzoic acid with a different kind of dash than the typesetter used, and the fourth complaint is that the results come back in primary-key order, which nobody has ever wanted.

You can push a fair distance further than that — full-text indexes in SQL Server or Postgres are real search engines and I have shipped features on both. But the moment the tokenisation rules are domain-specific, you need to own the analysis chain, and owning the analysis chain is the thing Lucene is for. This is a note about what stepping onto it actually buys, written after two years of doing exactly that on a chemistry corpus.

What a chemical name does to a normal tokeniser

Here is StandardAnalyzer, which is the sensible default for English prose, meeting four real queries:

input                          StandardAnalyzer produces
2-(acetyloxy)benzoic acid  →   2, acetyloxy, benzoic, acid
C9H8O4                     →   c9h8o4
α-D-glucose                →   α, d, glucose
50-78-2                    →   50, 78, 2

Every line is wrong in a different way. The first threw away the structure that distinguishes one isomer from another. The second is technically intact but lowercased into a token that will never match a document where the same formula was written C₉H₈O₄. The third split a stereodescriptor off the head of the name and left a bare d that now matches half the corpus. The fourth turned a CAS registry number into three integers.

The naive fix is a pile of regular expressions in front of the query box. That is the wrong place: whatever you do to the query, you must have already done to the document, or the two strings will never meet. Which is the single most useful thing Lucene teaches, and it teaches it structurally.

The inverted index is the primitive worth having

Underneath the analysers, Lucene is one data structure and it is not complicated to describe: for every term, the sorted list of documents containing it, with positions and offsets. benzoic → documents 4, 17, 903, at these word positions, at these character offsets.

That shape is what makes search cheap and ranked rather than expensive and unordered. A scan compares the query against every row. An inverted index intersects a handful of short lists and then scores only the documents that survived. Phrase queries fall out of the positions. Highlighting falls out of the offsets. Ranking has somewhere to stand, because the index already knows the term's document frequency.

What I did not appreciate until I had to read the internals: almost all of Lucene's cleverness is in making that structure small and sequential on disk. The term dictionary is an FST. Postings are delta-encoded and bit-packed with skip lists over them. Numerics live in a BKD tree so a range query does not become a term query with ten thousand clauses. Segments are immutable and get merged in the background, which is how writes and reads stop fighting.

That is the list of things I would have had to write badly. It is also the answer to "why not just build an index in Postgres" — you can, and it is a good index, but you do not get to replace the tokeniser with one that understands stereodescriptors.

Analysis is the part you actually write

An Analyzer is a factory for a chain: character filters, then a tokeniser, then token filters. Ours, roughly:

final class ChemicalAnalyzer extends Analyzer {

  private final NormalizeCharMap symbols;      // α → alpha, ′ → ', ‐ ‑ ‒ – → -
  private final SynonymMap synonyms;           // curated, from the substance registry

  @Override
  protected Reader initReader(String field, Reader reader) {
    // Character filters run before tokenisation and, crucially, keep an
    // offset map so highlighting still points at the original text.
    return new MappingCharFilter(symbols, reader);
  }

  @Override
  protected TokenStreamComponents createComponents(String field) {
    Tokenizer source = new WhitespaceTokenizer();

    TokenStream out = new WordDelimiterGraphFilter(
        source,
        WordDelimiterGraphFilter.GENERATE_WORD_PARTS
            | WordDelimiterGraphFilter.GENERATE_NUMBER_PARTS
            | WordDelimiterGraphFilter.CATENATE_ALL
            | WordDelimiterGraphFilter.PRESERVE_ORIGINAL,
        null);

    out = new LowerCaseFilter(out);
    out = new SynonymGraphFilter(out, synonyms, /* ignoreCase */ true);
    out = new FlattenGraphFilter(out);          // index side only — see below
    out = new KeywordRepeatFilter(out);         // keep the unstemmed form too
    out = new PorterStemFilter(out);
    out = new RemoveDuplicatesTokenFilter(out);

    return new TokenStreamComponents(source, out);
  }
}

Five details in there took real time to learn, so they are worth writing down.

PRESERVE_ORIGINAL with CATENATE_ALL is the whole trick for names with punctuation. 2-(acetyloxy)benzoic indexes as the parts, as the concatenation 2acetyloxybenzoic, and as itself. A user who types any of the three finds the document.

FlattenGraphFilter belongs on the index side and nowhere else. Both graph filters can emit a token lattice — one input position producing alternatives of different lengths. The index format cannot store a lattice, so you flatten it, accepting a slightly lossy result. At query time you must not flatten: the query parser handles the graph properly and builds the right disjunction, and flattening there is how you get phrase queries that silently stop matching.

KeywordRepeatFilter before the stemmer, RemoveDuplicates after. Every token is indexed twice — stemmed and not — so an exact form outranks a stemmed one without a second field. Two lines, and it removed a whole category of complaint.

Symbol normalisation is a character filter, not a token filter. It has to happen before tokenisation because it changes what the tokeniser sees, and MappingCharFilter keeps the offset correction so a highlight still lands on the right character in the source document. A hand-rolled String.replace in front of the analyser loses that and your highlights drift by one character per substitution.

Formulas and registry numbers do not go through any of this. They are a separate, unanalysed field:

doc.add(new StringField("formula", "C9H8O4", Field.Store.YES));   // exact, TermQuery
doc.add(new StringField("cas", "50-78-2", Field.Store.YES));
doc.add(new TextField("name", name, Field.Store.YES));            // analysed
doc.add(new TextField("body", body, Field.Store.NO));

StringField is one token, verbatim. Trying to make one clever analyser serve both prose and identifiers is the mistake I watched us nearly make twice; the answer is always another field. PerFieldAnalyzerWrapper then keeps the per-field choices in one place.

The rule that costs the most to learn

Index-time and query-time analysis have to agree, and when they do not, the symptom is never an error. It is a document that obviously matches and does not come back.

Two tools make that a ten-minute problem instead of an afternoon. The first is dumping the chain by hand, which we wrapped in a test helper:

static List<String> terms(Analyzer analyzer, String field, String text) throws IOException {
  var out = new ArrayList<String>();
  try (TokenStream ts = analyzer.tokenStream(field, text)) {
    var term = ts.addAttribute(CharTermAttribute.class);
    ts.reset();
    while (ts.incrementToken()) out.add(term.toString());
    ts.end();
  }
  return out;
}

@Test
void formula_survives_analysis() {
  assertThat(terms(indexAnalyzer, "name", "2-(acetyloxy)benzoic acid"))
      .contains("2acetyloxybenzoic", "2", "acetyloxy", "benzoic", "acid");
}

We ended up with about ninety of those assertions, and they are the highest value tests in the search tier. They are also the ones a reindex will break loudly, which is exactly what you want.

The second is explain:

System.out.println(searcher.explain(query, docId));

Which prints the whole scoring tree — which clauses matched, each term's IDF, the length normalisation, the final sum. Nobody's intuition about relevance survives contact with an Explanation, including mine. Half the times I was sure the ranking was broken, the ranking was fine and my query had a clause I did not intend.

The extension points that made it ours

This is the actual argument for building on Lucene rather than on a search product: every layer is replaceable, and I mean the layers you would not expect.

• Similarity. BM25 is the default and is the right default. We tuned k1 and b per field — new BM25Similarity(1.2f, 0.35f) on name, because length normalisation is close to meaningless on a field where every value is a substance name, and left body alone. A PerFieldSimilarityWrapper holds that.
• Payloads. The annotator emits a confidence per annotation, and it goes into the postings as a payload on the term, then into the score through a PayloadScoreQuery. A term the NLP pipeline was unsure about contributes less. There is nowhere to put that in a LIKE query and nowhere obvious to put it in most search products.
• Intervals. Intervals.ordered(Intervals.term("acetylsalicylic"), Intervals.maxgaps(4, Intervals.term("synthesis"))) expresses this name, then that word, within five positions — a proximity language with real composition, and much easier to reason about than the spans queries it replaced.
• DocValues. A column-oriented per-document store, which is how you sort and facet without loading fields. Publication year as a NumericDocValues, substance class as SortedSetDocValuesFacetField.
• UnifiedHighlighter. Given offsets in the postings, it re-runs the query against a single document to produce snippets with the matched spans marked. It reuses the analysis chain, so a synonym hit highlights the word that was actually in the document.
• SearcherManager. Near-real-time search: an IndexSearcher is a snapshot, and the manager hands out the current one with reference counting so a reopen does not pull the index out from under a running query. Newly annotated documents appear in seconds without a restart.

None of that is exotic and all of it is in the box. The number of afternoons that has saved me is the reason I keep listing Lucene under things I am interested in rather than things I once used.

Where it is the wrong answer

Lucene is a library, and the failure mode of a library this capable is reaching for it as if it were a database.

It is not a database. One index, one IndexWriter, one process. No transactions spanning documents, no queries across indexes, no replication. Durability is a commit, which is an fsync and not cheap, so you batch — and if the process dies between commits, that batch is gone and something upstream must be able to replay it. Our source of truth was Virtuoso and the index was derived, always, deliberately.

Changing analysis means reindexing. Everything above is baked into the terms on disk. Adding one synonym rule to the analyser means every affected document is wrong until it is rebuilt. Budget for a full rebuild being routine and automated on day one, not a thing you discover in month four.

Distribution is not included. Sharding, replication, failover, a query API, cluster state — that is precisely the gap Solr and Elasticsearch fill, and if you need those, use them rather than growing your own. Our corpus fit on one machine's index with room to spare, which is the only reason straight Lucene was defensible.

Wildcards and regexes will eat you. A leading wildcard has to enumerate the term dictionary. RegexpQuery is worse. If users need infix matching, index n-grams at index time and pay for it there, in disk, where you can measure it.

And if you are on .NET, know what you are signing up for. Lucene.NET is a careful, faithful port and I have shipped on it, but the current release line is still the 4.8.0 betas — years of Java-side improvements not yet available, and a beta version number you have to explain to somebody. On this product it settled the question of which runtime the search tier lived on: the index was Java because Lucene is Java, and the annotator stayed .NET for reasons I wrote about a fortnight ago. Two runtimes, each one where its best library already was.

The honest summary

I did not choose Lucene because I like Java. I chose it because the list of things I would otherwise have had to write — an FST term dictionary, a bit-packed postings format with skip lists, a BKD tree, a merge policy, crash-safe commits, BM25, a highlighter that agrees with the analyser — is a career, and none of it is the thing anybody is paying for.

What is left after you step onto it is the part that is genuinely yours: the analysis chain, the fields, the synonym list, and ninety tests asserting that a chemical name still looks like itself after normalisation. That is a couple of weeks of work rather than a couple of years, and it is the couple of weeks where the domain knowledge actually lands.