Vishesh Duggar bio photo

Vishesh Duggar

CTO Software @vamstar. 15 years in tech. Entrepreneurial with a keen interest in product development and usability. Interested in working on high impact ideas.

Subscribe to my mailing list

Are you looking for a tech advisor or a CTO?

Contact on LinkedIn

Vamstar LinkedIn Twitter Github

I am currently working on project that involves dealing with massive amounts of data. To a point where we are seeing performance issues on the client.

One of the issues we’ve identified that can be improved includes what we send to the UI. We are indexing public documents and web pages for certain industry. For a job like this you don’t just store data that you show but also index analytical data. Also, our data model is pretty big; most users will almost never try to see all the fields together.

To reduce the size of the network response we have started sending only a subset of fields. It’s fairly standard to use query projections or field projections in databases but doing so with a search index is fairly easy yet not so common. ElasticSearch chooses to refer this as Source Filtering.

You can simply pass the field(s) you want to returned in the query as _source.

Turn off getting the stored document all together by passing false

GET /_search
{
    "_source": false,
    "query" : {
        "term" : ...
    }
}

Get a subset of fields using a wildcard for example if there is a field with nested fields called address

GET /_search
{
    "_source": "address.*",
    "query" : {
        "term" : ...
    }
}

Get a subset of explicitly named fields by sending in the exact names

GET /_search
{
    "_source": [ "name", "address", "bio"],
    "query" : {
        "term" : ...
    }
}

If you want to do fine tune further there is a way to pass both include and exclude list

GET /_search
{
    "_source": {
      "includes": [ "name", "address", "bio"],
      "excludes": [ "email"]
    }
    "query" : {
        "term" : ...
    }
}

It was easy enough to figure out but more importantly the lesson was to never forget to pay attention to what you are sending to the client.

Close to 15 years in tech; I've served as a CTO and advisor to multiple organizations. Brought close to 20 products to market. As a founding member of multiple organizations I've done everything from tech to stratgey, sales, marketing, hiring, accounting and more. Experience in a variety of technologies including but not limited to AWS, Node, React, Serverless, ElasticSearch, Groovy, Java, Typescript, Angular, Grails, PHP, Drupal, Wordpress.

Always interested in looking at new tech, strategy and ways I can add value to organizations.

[Contact me.](https://www.linkedin.com/in/experienced-cto/).