Full-text search on AWS with OpenSearch, Lambda and CDK (step by step)

Users type "funny cat" into a search box and expect every video whose title, description or tags mention it, ranked sensibly. DynamoDB can't do that: it finds items by key, not by words inside text. Scanning the whole table and filtering in code works for 100 items and collapses at 100,000.

This is a job for a search engine. On AWS that's Amazon OpenSearch Service (the open-source fork of Elasticsearch). Here's how it works and how we wire it into the serverless VidShare platform.

How full-text search works: the inverted index

A database stores rows and looks them up by ID. A search engine also builds an inverted index: for every word, the list of documents that contain it.

Search for "funny cat" and OpenSearch looks up funny → [1, 2] and cat → [1], then scores the matches. Video 1 contains both words, so it ranks first.

Think of a library. Finding "the book titled Hamlet" is easy if books are shelved by title. Finding "the book that contains to be, or not to be" needs an index of the words inside the pages.

Clusters, nodes and indices

• A domain (cluster) is one or more nodes, each a server running OpenSearch. One node is elected master to coordinate.
• Data lives in indices. An index is roughly a collection of similar documents, like video.
• Documents are JSON. A mapping tells OpenSearch how to treat each field: text fields get analysed into words for full-text search.

Talk to it: REST and the Query DSL

OpenSearch is a REST API. Searching an index is a GET to /<index>/_search with a JSON body written in the Query DSL:

GET /video/_search
{
  "query": {
    "multi_match": {
      "query": "funny cat",
      "fields": ["title", "description", "tags"]
    }
  }
}

• term-level queries (term) match exact values, like a status or an ID.
• full-text queries (match, multi_match) analyse your search text into words and match them against analysed fields.

Create the domain with CDK

import { Domain, EngineVersion } from "aws-cdk-lib/aws-opensearchservice";
 
const domain = new Domain(this, "Domain", {
  version: EngineVersion.OPENSEARCH_2_3,
  capacity: { dataNodes: 1, dataNodeInstanceType: "t3.small.search" },
  removalPolicy: cdk.RemovalPolicy.DESTROY,
});
 
domain.grantReadWrite(videoCrudHandler);          // the Lambda that searches
domain.grantReadWrite(mediaConvertEventHandler);  // the Lambda that indexes

Lambdas sign their requests with their IAM role, so grantReadWrite is all they need. No passwords in environment variables.

The index problem, and custom resources

Deploying the stack creates the domain, but not the video index with its mapping. You'd have to create that by hand after every deploy, from Postman or the console.

CDK's answer is a custom resource: a Lambda that CloudFormation calls on stack create, update and delete, so you can run any setup code as part of the deployment.

The handler from the course creates the index only on Create:

import { CdkCustomResourceHandler } from "aws-lambda";
import { OpensearchClient } from "../../lib/opensearch-client";
 
export const handler: CdkCustomResourceHandler = async (event) => {
  if (event.RequestType === "Create") {
    const client = new OpensearchClient({
      endpoint: event.ResourceProperties.endpoint,
      region: event.ResourceProperties.region,
    });
    await client.indices.create({
      index: event.ResourceProperties.indexName,
      body: event.ResourceProperties.indexProperties,
    });
  }
  return { PhysicalResourceId: event.ResourceProperties.indexName };
};

And in the stack, the index name and mapping are passed as properties:

new cdk.CustomResource(this, "OpensearchVideoIndex", {
  serviceToken: new crProvider.Provider(this, "CustomResourceLambdaProvider", {
    onEventHandler: opensearchIndexCreatorLambda,
  }).serviceToken,
  properties: {
    endpoint: `https://${domain.domainEndpoint}`,
    region: this.region,
    indexName: "video",
    indexProperties: {
      mappings: {
        properties: {
          title: { type: "text" },
          description: { type: "text" },
          tags: { type: "text" },
        },
      },
    },
  },
});

If the Lambda throws, CloudFormation rolls the deployment back. Now cdk deploy produces a fully working search, every time, in any region.

Index and search from Lambda

A small class keeps the OpenSearch details out of the handlers:

export class OpenSearchIndex {
  constructor(private indexName: string, private client: OpensearchClient) {}
 
  put({ doc, id }: { doc: any; id: string }) {
    return this.client.index({ id, body: doc, index: this.indexName, refresh: true });
  }
 
  async search(keyword: string, fields: string[]) {
    const res = await this.client.search({
      index: this.indexName,
      body: { query: { multi_match: { query: keyword, fields } } },
    });
    return res.body.hits.hits.map((v: any) => ({ id: v._id, ...v._source }));
  }
}
 
// usage
videoIndex.search("funny cat", ["title", "description", "tags"]);

GET /video?search=cat on the API now calls searchVideo, and the test for it uses the same spy-and-mock pattern as every other handler (see TDD for AWS Lambda with Jest).

Add search to a real platform

VidShare ends with this: a serverless video platform where uploads are transcoded, stored, and searchable by title, description and tags. 109 lessons, all in TypeScript with tests, deployed with CDK. Watch the free architecture lesson first.