Search Your Data
This guide will show you how to search your data without decrypting it.
Key concepts in this guide:
Getting Started
To get started, you will need a Basis Theory account.
Next you will need a Private Application in order to create and search tokens.
Click here to create a Private Application or login to your Basis Theory account and create a new application with the following settings:
- Name - Search Data Guide
- Application Type - Private
- Permissions -
token:create
,token:search
Create Tokens
When you create a token, you can supply one or more Search Indexes, which are Expressions based on the Liquid template language. These Search Indexes are computed at the time of token creation to securely generate search indexes needed to enable searching.
Run the following in your terminal to create some tokens:
curl "https://api.basistheory.com/tokenize" \
-X "POST" \
-H "BT-API-KEY: <API_KEY>" \
-H "Content-Type: application/json" \
-d '[{
"type": "token",
"data": {
"first_name": "Luke",
"last_name": "Skywalker",
"social_security_number": "111-22-3333",
"email_address": "luke@skywalkerranch.com"
},
"mask": {
"first_name": "{{ data.first_name }}",
"last_name": "{{ data.last_name | slice: 0 }}",
"social_security_number": "{{ data.social_security_number | reveal_last: 4 }}",
"email_address": "{{ data.email_address | split: '\''@'\'' | last }}"
},
"metadata": {
"customer_type": "vip",
"spaceship": "X-Wing"
},
"containers": [
"/customers/skywalkers/"
],
"search_indexes": [
"{{ data.first_name | downcase }}",
"{{ data.last_name | downcase }}",
"{{ data.social_security_number }}",
"{{ data.social_security_number | last4 }}",
"{{ data.email_address | downcase }}",
"{{ data.email_address | split: '\''@'\'' | last }}"
]
}, {
"type": "token",
"data": {
"first_name": "Han",
"last_name": "Solo",
"social_security_number": "444-55-6666",
"email_address": "han.solo@skypilots.com"
},
"mask": {
"first_name": "{{ data.first_name }}",
"last_name": "{{ data.last_name | slice: 0 }}",
"social_security_number": "{{ data.social_security_number | reveal_last: 4 }}",
"email_address": "{{ data.email_address | split: '\''@'\'' | last }}"
},
"metadata": {
"customer_type": "vip",
"spaceship": "Millenium Falcon"
},
"containers": [
"/customers/solo/"
],
"search_indexes": [
"{{ data.first_name | downcase }}",
"{{ data.last_name | downcase }}",
"{{ data.social_security_number }}",
"{{ data.social_security_number | last4 }}",
"{{ data.email_address | downcase }}",
"{{ data.email_address | split: '\''@'\'' | last }}"
]
}, {
"type": "token",
"data": {
"first_name": "Leia",
"last_name": "Organa",
"social_security_number": "777-88-9999",
"email_address": "leia@skywalkerranch.com"
},
"mask": {
"first_name": "{{ data.first_name }}",
"last_name": "{{ data.last_name | slice: 0 }}",
"social_security_number": "{{ data.social_security_number | reveal_last: 4 }}",
"email_address": "{{ data.email_address | split: '\''@'\'' | last }}"
},
"metadata": {
"customer_type": "standard",
"spaceship": "Tantive IV"
},
"containers": [
"/customers/skywalkers/"
],
"search_indexes": [
"{{ data.first_name | downcase }}",
"{{ data.last_name | downcase }}",
"{{ data.social_security_number }}",
"{{ data.social_security_number | last4 }}",
"{{ data.email_address | downcase }}",
"{{ data.email_address | split: '\''@'\'' | last }}"
]
}]'
<API_KEY>
with the Private API Key you created in the Getting Started step.This will create several search indexes:
- Lowercase first name
- Lowercase last name
- Full social security number
- Last four of the social security number
- Lowercase email address
- Domain of the email address
Search Tokens
Now that we have some data tokenized, we can demonstrate several possible ways to search over the data. Basis Theory uses a Lucene-based query syntax to power the search engine.
Search by Data
Run the following command in your terminal to search for tokens having the indexed data value 3333
,
which we expect to match the last 4 digits of the social_security_number
field on Luke Skywalker's token:
curl "https://api.basistheory.com/tokens/search" \
-H "BT-API-KEY: <API_KEY>" \
-H "Content-Type: application/json" \
-X "POST" \
-d '{
"query": "data:3333",
"page": 1,
"size": 20
}'
You should see a JSON response similar to:
{
"pagination": {
"total_items": 1,
"page_number": 1,
"page_size": 20,
"total_pages": 1
},
"data": [
{
"id": "5855f821-a207-4f07-8204-4b7c96cb2bbc",
"type": "token",
"tenant_id": "dd19bffb-db97-4ef7-8630-54834a7c1243",
"data": {
"first_name": "Luke",
"last_name": "S",
"social_security_number": "XXX-XX-3333",
"email_address": "skywalkerranch.com"
},
"metadata": {
"customer_type": "vip",
"spaceship": "X-Wing"
},
"created_by": "f1d47dd9-34ac-407c-bb0c-99ad786f9167",
"created_at": "2022-12-21T20:49:11.6884577+00:00",
"fingerprint": "4wPsbGizTkqeh3ZGXFEbPep5zmf6ffokX37SDEAw9m2S",
"fingerprint_expression": "{{ data | stringify }}",
"mask": {
"first_name": "{{ data.first_name }}",
"last_name": "{{ data.last_name | slice: 0 }}",
"social_security_number": "{{ data.social_security_number | reveal_last: 4 }}",
"email_address": "{{ data.email_address | split: '@' | last }}"
},
"search_indexes": [
"{{ data.social_security_number }}",
"{{ data.email_address | split: '@' | last }}",
"{{ data.social_security_number | last4 }}",
"{{ data.last_name | downcase }}",
"{{ data.email_address | downcase }}",
"{{ data.first_name | downcase }}"
],
"containers": [
"/customers/skywalkers/"
]
}
]
}
Search by Metadata
Run the following command in your terminal to search for tokens having a metadata key customer_type
with a value of vip
:
curl "https://api.basistheory.com/tokens/search" \
-H "BT-API-KEY: <API_KEY>" \
-H "Content-Type: application/json" \
-X "POST" \
-d '{
"query": "metadata.customer_type:vip",
"page": 1,
"size": 20
}'
You should see a JSON response similar to:
{
"pagination": {
"total_items": 2,
"page_number": 1,
"page_size": 20,
"total_pages": 1
},
"data": [
{
"id": "400394da-79d3-41f6-a45f-4e60c3e5b0a0",
"type": "token",
"tenant_id": "dd19bffb-db97-4ef7-8630-54834a7c1243",
"data": {
"first_name": "Han",
"last_name": "S",
"social_security_number": "XXX-XX-6666",
"email_address": "skypilots.com"
},
"metadata": {
"customer_type": "vip",
"spaceship": "Millenium Falcon"
},
"created_by": "f1d47dd9-34ac-407c-bb0c-99ad786f9167",
"created_at": "2022-12-21T20:49:11.6884577+00:00",
"fingerprint": "Hi9TgzHi2otCqBeHKM5v9eAroGtUGUzTPFUCX3akRAEM",
"fingerprint_expression": "{{ data | stringify }}",
"mask": {
"first_name": "{{ data.first_name }}",
"last_name": "{{ data.last_name | slice: 0 }}",
"social_security_number": "{{ data.social_security_number | reveal_last: 4 }}",
"email_address": "{{ data.email_address | split: '@' | last }}"
},
"search_indexes": [
"{{ data.social_security_number | last4 }}",
"{{ data.last_name | downcase }}",
"{{ data.email_address | downcase }}",
"{{ data.social_security_number }}",
"{{ data.email_address | split: '@' | last }}",
"{{ data.first_name | downcase }}"
],
"containers": [
"/customers/solo/"
]
},
{
"id": "5855f821-a207-4f07-8204-4b7c96cb2bbc",
"type": "token",
"tenant_id": "dd19bffb-db97-4ef7-8630-54834a7c1243",
"data": {
"first_name": "Luke",
"last_name": "S",
"social_security_number": "XXX-XX-3333",
"email_address": "skywalkerranch.com"
},
"metadata": {
"customer_type": "vip",
"spaceship": "X-Wing"
},
"created_by": "f1d47dd9-34ac-407c-bb0c-99ad786f9167",
"created_at": "2022-12-21T20:49:11.6884577+00:00",
"fingerprint": "4wPsbGizTkqeh3ZGXFEbPep5zmf6ffokX37SDEAw9m2S",
"fingerprint_expression": "{{ data | stringify }}",
"mask": {
"first_name": "{{ data.first_name }}",
"last_name": "{{ data.last_name | slice: 0 }}",
"social_security_number": "{{ data.social_security_number | reveal_last: 4 }}",
"email_address": "{{ data.email_address | split: '@' | last }}"
},
"search_indexes": [
"{{ data.social_security_number }}",
"{{ data.email_address | split: '@' | last }}",
"{{ data.social_security_number | last4 }}",
"{{ data.last_name | downcase }}",
"{{ data.email_address | downcase }}",
"{{ data.first_name | downcase }}"
],
"containers": [
"/customers/skywalkers/"
]
}
]
}
Perform Compound Search
Run the following command in your terminal to search for tokens having both a metadata key customer_type
with a value of vip
and a metadata key spaceship
with a value of X-Wing
:
curl "https://api.basistheory.com/tokens/search" \
-H "BT-API-KEY: <API_KEY>" \
-H "Content-Type: application/json" \
-X "POST" \
-d '{
"query": "metadata.customer_type:vip AND metadata.spaceship:X-Wing",
"page": 1,
"size": 20
}'
You should see a JSON response similar to:
{
"pagination": {
"total_items": 1,
"page_number": 1,
"page_size": 20,
"total_pages": 1
},
"data": [
{
"id": "5855f821-a207-4f07-8204-4b7c96cb2bbc",
"type": "token",
"tenant_id": "dd19bffb-db97-4ef7-8630-54834a7c1243",
"data": {
"first_name": "Luke",
"last_name": "S",
"social_security_number": "XXX-XX-3333",
"email_address": "skywalkerranch.com"
},
"metadata": {
"customer_type": "vip",
"spaceship": "X-Wing"
},
"created_by": "f1d47dd9-34ac-407c-bb0c-99ad786f9167",
"created_at": "2022-12-21T20:49:11.6884577+00:00",
"fingerprint": "4wPsbGizTkqeh3ZGXFEbPep5zmf6ffokX37SDEAw9m2S",
"fingerprint_expression": "{{ data | stringify }}",
"mask": {
"first_name": "{{ data.first_name }}",
"last_name": "{{ data.last_name | slice: 0 }}",
"social_security_number": "{{ data.social_security_number | reveal_last: 4 }}",
"email_address": "{{ data.email_address | split: '@' | last }}"
},
"search_indexes": [
"{{ data.social_security_number }}",
"{{ data.email_address | split: '@' | last }}",
"{{ data.social_security_number | last4 }}",
"{{ data.last_name | downcase }}",
"{{ data.email_address | downcase }}",
"{{ data.first_name | downcase }}"
],
"containers": [
"/customers/skywalkers/"
]
}
]
}
Conclusion
Utilizing Token Search empowers you to provide business functionality against your encrypted data without the need to decrypt it, enabling things like reporting and customer service operations.