Bulk data downloads enable users to download scholarly and patent bulk data files via the API or UI. The API allows users to track the availability of new bulk data files from Lens and automate the process of downloading and keeping data up-to-date. To use the bulk data downloads API, you will need an access token, which can be generated in the Subscriptions tab of your Lens account.
Note: The Lens allows the same access token to be used across all API products.
File Format
Bulk data download files are structured in JSON Lines format, where each line
contains a separate JSON document representing an individual record. Files are compressed in .gz
format. The data schema for records is the same as the search API
response schema. Sample bulk data files can be downloaded here:
Using the API
The access token can be submitted using the token request parameter, or alternatively in the
Authorization field of the HTTP request header ( see Getting Started > API Access in the API
documentation for details).
Latest Data Release Endpoint
This endpoint is useful for getting the latest bulk data file listing and associated metadata. Since the bulk data files are released fortnightly, these endpoints can be used in a scheduler to automate the availability check for new files.
https://api.lens.org/bulk/scholarly/releasehttps://api.lens.org/bulk/patent/release
The fields and data schema used in the Release Endpoint are listed below.
| Field | Description | Example |
| bulkDataProduct | Product associated with download file | SCHOLARLY, PATENT |
| dataVersion | Bulk data version (usually YEAR-WEEK format) | 202336 |
| precursorDataVersion | The previous DataVersion to the current file | 202334 |
| records | Number of records in the file | 264329940 |
| md5sum | MD5 Hash of the compressed file. N.B. This field will be deprecated in future, please use the checksums field instead. |
de9acdacff6e099d5e6c7aa22d0eaeac |
| checksums | Hash of the compressed file with various hash functions. | {"sha1":"272faafb8e46aba9c934155e75b69247ff42731b", "md5": "371639385ab44f4c89e6b518908ebd53", ...} |
| fileName | Name of the file (usually product-week-UUID.jsonl.gz) | scholarly-202336-0bd17e86-8bed-4c9b-a863-9f3928e4c71b.jsonl.gz |
| firstRecordId | LensId of the first record in the file | 043-486-543-492-05X |
| lastRecordId | LensId of the last record in the file | 145-007-429-245-212 |
| dateProcessed | Date the file was processed by Lens | 2023-09-16 |
| downloadAccessKey | Access key to download the file using the Download Endpoint | hN74L3upIqYNU4qK-sdU7-g1YwakLw1Hu3edL6wa7YP3cS... |
| size | Compressed size in bytes | 204149739824 |
| rawSize | Actual size of the file in bytes after uncompressing | 716653744007 |
Example Request:
[GET] https://api.lens.org/bulk/patent/release?token={access_token}
Example Response:
{
"bulkDataProduct": "PATENT",
"dataVersion": "202338",
"records": 150757805,
"md5sum": "c1fe56f4dd327823f3be9294efc6b2a8",
"firstRecordId": "054-211-414-158-561",
"lastRecordId": "168-549-513-683-443",
"dateProcessed": "2023-09-29",
"downloadAccessKey": "5jJnJVadp-wAFweuaGN_XRMVinSNA5nP4TM1cOjpUUrIMvTaqb68...",
"size": 458139434490
}
Delta Releases Endpoint
Lens bulk data is also available in Back-file (snapshot) and Delta-files format. The Back-file is generated once a year and includes
all previous records at the time of generation. For subsequent updates, the Delta-files are released on a weekly (patents) / fortnightly
(scholarly works) basis and include the New, Updated and Deleted records since the back-file or
previous delta-file was generated, which can be applied on top of the Back-file to keep the bulk data updated. The Releases Endpoints are:
https://api.lens.org/bulk/scholarly/releaseshttps://api.lens.org/bulk/patent/releases
The fields used in the Releases Endpoint are the same as the fields used in the Release Endpoint, however
the snapshot Back-file and delta files are listed in separate objects, e.g.
{
"snapshot": {...},
"delta": [{...},{...}]
}
Note: you can also use the Release Endpoint to get the latest bulk data snapshot for use as the Back-file to which subsequent Delta-files can be applied to keep it up-to-date. For the updates, you can download and apply the Delta-files to the Back-file starting from the next data version after the Back-file data version.
Example Request:
[GET] https://api.lens.org/bulk/patent/releases?token={access_token}
Example response:
{
"snapshot": {
"bulkDataProduct": "PATENT",
"dataVersion": "202418",
"records": 1000000,
"checksums": {
"crc32c": "BK2+7g==",
"crc32": "Q+heng==",
"sha1": "6e0d4185297daaff502b089de7bb9d23f2268b16",
"sha256": "1a2e4a3bd183073436e2f4266d3e9f29743dd72b8c6b541afa85c29dfb3963ff",
"md5": "cea2645e4796e1dc50ea763ba01c64e5"
},
"fileName": "demo-patent-202418-20cc8f77-c744-4001-aaf7-97b857abe43a.jsonl.gz",
"firstRecordId": "000-800-002-831-656",
"lastRecordId": "000-166-158-262-298",
"dateProcessed": "2024-05-15",
"downloadAccessKey": "Xs3q0uyJ2dXT93o81DXOuXEVDe82gcfacfbqJYxmCjJqfkp9_tBoenPmBCkvVugJOo6fBSqEY4UdFA8YSfywxvUfVZ-uJXWg11hHnGUQOyxHNARHLRoDoKsHr04As2-soFmSU5YKq_FpAJBBaXhf2LEn_P29VNW6G06jltmH6YXSoi1Qg00ZpK-LkK8bQIfhC07KLzJw7pSzp3BmvCPRCdLApHd3J9a5uXjBmWegPDKd3d9ZlxJuY9C41gB-sRhNcuym3NNIHVXQv94uERgpYg",
"size": 3219334950,
"rawSize": 15409610761,
"prefix": "demo-"
},
"delta": [
{
"bulkDataProduct": "PATENT",
"dataVersion": "202419",
"records": 10000,
"checksums": {
"crc32c": "PIU6Xg==",
"crc32": "ELIj2Q==",
"sha1": "866d3f84f3fe5a996d3b5836e8562384dc925568",
"sha256": "cfc9fee941352b4e4efa09e59e88f7a9ca536dca2a29d2209176c9d6f7655601",
"md5": "a7fc4bdecebcf573a0b8d898a6262717"
},
"fileName": "demo-patent-202419-b76536b4-cf5f-491d-8645-c8c479cd0ce9.jsonl.gz",
"firstRecordId": "000-600-020-657-480",
"lastRecordId": "000-228-080-973-567",
"dateProcessed": "2024-05-16",
"downloadAccessKey": "E3v18g2g7s2Ctb2XFnbfZU1EH2b9_vPXxFadYHKGFUjYu3K_ANjVX5w9RJfzl3Q4Ui94V5x8RLInhQDyEEjCx8DT5BO3pEej3Q8q3_2kIq_Erm2vg15Ffv0uXda-MyVmnblqaf2NxowWaKO0xVsf9LPbkAY_ZeaynlXFXPajriBSqbElwa5-3qFRiSJDOzz1W4AUv_hR_9x8xmSOY7NPTvxXOwWp5BzRIDyoS3wcCkGvqnu41xSd9dK3k8doOXu3AafJmo5VLGSNMbJ9q0HJsQ",
"size": 93524952,
"rawSize": 459175370,
"prefix": "demo-"
}
]
}
Download Endpoint
Bulk data files can be downloaded using either the API or UI. Downloads via the API and UI are both subject to rate-limiting based on your subscription plan.
Download using API
To download bulk data files using the API, you will need the file download access key and your API access token. The
download access key (downloadAccessKey) is available from the Release Endpoint listed above. The
download API endpoint is:
https://api.lens.org/bulk/download/{downloadAccessKey}
The integrity of the downloaded file can be verified using the md5sum available from the Release Endpoint. Similarly, the number of records and first/last record LensId can be used to verify the file after extraction.
Note: Take note of the uncompressed file size from the Release Endpoint to ensure you have enough disk space
if you are extracting it. Also, whatever HTTP client you use will need to be able to follow 302
redirects. E.g.
wget 'https://api.lens.org/bulk/download/{downloadAccessKey}?token={access_token}' -O filename.gz
Where {downloadAccessKey} is from the Release Endpoint and {access_token} is your Lens
subscription access token.
Example Scripts
Download Automation
The below Python script can be used to automate the download of the latest bulk data file to a location. The script allows users to implement storage of the download file information from the Latest Data Release Endpoint and periodically perform a check for newer files and downloads new files to a specified location.
import requests
from datetime import datetime
import os
import hashlib
api_host = 'https://api.lens.org'
api_token = 'YOUR TOKEN'
output_location = os.getcwd()
class ReleaseFileInfo:
def __init__(self, product, data_version, num_records, checksums, file_name, first_record_id, last_record_id,
date_processed, download_access_key, compressed_size, uncompressed_size):
self.product = product
self.data_version = data_version
self.num_records = num_records
self.checksums = checksums
self.file_name = file_name
self.first_record_id = first_record_id
self.last_record_id = last_record_id
self.date_processed = date_processed
self.download_access_key = download_access_key
self.compressed_size = compressed_size
self.uncompressed_size = uncompressed_size
def __get_current_release(data_type) -> ReleaseFileInfo:
release_url = api_host + '/bulk/%s/release' % data_type
print(release_url)
headers = {'Authorization': 'Bearer ' + api_token, 'Content-Type': 'application/json'}
release = requests.get(release_url, headers=headers).json()
return ReleaseFileInfo(release['bulkDataProduct'], release['dataVersion'], release['records'], release['checksums'],
release['fileName'], release['firstRecordId'], release['lastRecordId'],
datetime.strptime(release['dateProcessed'], '%Y-%m-%d'), release['downloadAccessKey'],
release['size'], release['rawSize'])
# Check if the provided download has been already processed
def __download_already_exists(release_info):
raise NotImplementedError('Implement release update file check')
# Persist the latest parse info in your store. Use the same to check against `__download_already_exists`
def __persist_latest_parse_info(release_info):
raise NotImplementedError('Implement persistence of downloaded file info')
# Download and check the file integrity
def __download_file(release_info):
download_url = api_host + '/bulk/download/' + release_info.download_access_key
headers = {'Authorization': 'Bearer ' + api_token}
output_filename = output_location + '/' + release_info.product + '/' + release_info.data_version + '/' + release_info.file_name
os.makedirs(os.path.dirname(output_filename), exist_ok=True)
with requests.get(download_url, headers=headers, stream=True) as r:
if r.status_code == requests.codes.too_many_requests:
print('Download is rate limited. Please check your usage')
return
with open(output_filename, 'wb') as f:
file_hash = hashlib.md5()
# iterate with r.iter_content(chunk_size=...) if you want to download file uncompressed.
chunk = r.raw.read(1024*1024) # chunks of 1MB, adjust as required
while chunk:
file_hash.update(chunk)
f.write(chunk)
chunk = r.raw.read(1024*1024)
file_md5 = file_hash.hexdigest()
release_md5 = release_info.checksums['md5']
if file_md5 != release_md5:
raise ValueError('File integrity check failed. Expected md5 (%s), downloaded md5 (%s)', release_md5, file_md5)
print('Finished downloading file: %s' % output_filename)
def start_updator(data_type):
release_info = __get_current_release(data_type)
if __download_already_exists(release_info):
print('This release has been already processed: %s(%s)' % (release_info.product, release_info.data_version))
else:
__download_file(release_info)
__persist_latest_parse_info(release_info)
# Usage
# start_updator('scholarly')
# start_updator('patent')
Data Import/Processing
This example Python script utilises a single producer with multiple consumers to make the data import process more performant. It is a simple processor implementation with a single producer to read the downloaded file and 3 consumers to process the records in parallel. You can tune the performance by increasing number consumer threads and size of the queue.
import gzip
import json
import queue
from threading import Thread
from queue import Empty
from time import sleep
def consume(data_queue, consumer_id):
while True:
try:
record = data_queue.get(block=False)
except Empty:
# wait for data to be available
sleep(0.5)
continue
if record is None:
break
# Do something with the data
print('%s: processing record > %s' % (consumer_id, json.loads(record)['lens_id']))
print('Completed: ' + consumer_id)
def produce(location, data_queue, consumers_count):
with gzip.open(location, 'rt') as f:
for record in f:
data_queue.put(record)
# poison pill to stop the consumers
for i in range(consumers_count):
data_queue.put(None)
num_consumers = 3
file_location = 'scholarly-202346-6f64a086-94a7-47bc-b21d-5cec945e3705.jsonl.gz'
record_queue = queue.Queue(maxsize=100)
# Start multiple consumers
consumers = [Thread(target=consume, args=(record_queue, 'consumer-%s' % n,)) for n in range(num_consumers)]
for consumer in consumers:
consumer.start()
# Single producer to read the file and push record into the queue
producer = Thread(target=produce, args=(file_location, record_queue, len(consumers),))
producer.start()
# Finish the processing pipeline
producer.join()
for consumer in consumers:
consumer.join()
Batch Download and Filtering
The below example script uses Shell utilities to download the bulk data file and and split it on the fly into smaller chunks of approximately 1 Gigabyte each and compresses them using zstd. You can add a line filter anywhere before the split using a regexp, grep or a Python script, to filter out whatever you need, making the local storage requirements smaller. This assumes you’re using a device/environment where gnu split is available.
curl -L {download-url-with-token} | gunzip | split --line-bytes=1G --numeric-suffixes --suffix-length=6 --filter='zstd -5 > $FILE.jsonl.zst' - blk-
Credit: National Science Foundation | Dawid Weiss
Delta Updates
This example Python script utilises the Releases endpiont to download the latest delta-files and apply updates to the back-file so it is up to date. Note: The exact code will be dependent on your intenral database and environment.
import gzip
import json
from typing import Callable
import requests
from datetime import datetime
import os
api_host = 'https://api.lens.org'
api_token = 'YOUR TOKEN'
class ReleaseFileInfo:
def __init__(self, release):
self.product = release['bulkDataProduct']
self.data_version = release['dataVersion']
self.num_records = release['records']
self.md5_checksum = release['checksums']['md5']
self.file_name = release['fileName']
self.first_record_id = release['firstRecordId']
self.last_record_id = release['lastRecordId']
self.date_processed = datetime.strptime(release['dateProcessed'], '%Y-%m-%d')
self.download_access_key = release['downloadAccessKey']
self.compressed_size = release['size']
self.uncompressed_size = release['rawSize']
class Releases:
def __init__(self, releases_json):
self.snapshot = ReleaseFileInfo(releases_json['snapshot'])
self.delta = list(map(ReleaseFileInfo, releases_json['delta']))
# Get all available releases for data type (patent, scholarly)
def __get_all_releases(data_type) -> Releases:
releases_url = api_host + '/bulk/%s/releases' % data_type
headers = {'Authorization': 'Bearer ' + api_token, 'Content-Type': 'application/json'}
response = requests.get(releases_url, headers=headers)
return Releases(response.json())
# Get latest available bulk or delta release info.
def __get_latest_release(data_type, is_delta: bool) -> ReleaseFileInfo:
release_url = api_host + '/bulk/%s/release?delta=%s' % (data_type, str(is_delta).lower())
headers = {'Authorization': 'Bearer ' + api_token, 'Content-Type': 'application/json'}
response = requests.get(release_url, headers=headers)
if response.status_code == requests.codes.no_content:
raise RuntimeError('Cannot find any release for %s delta(%s)' % (data_type, is_delta))
return ReleaseFileInfo(response.json())
# Persist the latest parse info in your store. Use the same to check against `__download_already_exists`
def __persist_latest_parse_info(release_info):
raise NotImplementedError('Implement persistence of downloaded file info. This will be checked to confirm if the '
'file is already processed.')
# To avoid duplicate process, check if the release is already processed.
def __download_already_exists(release_info):
raise NotImplementedError('Implement check if release %s is already processed.' % release_info.data_version)
# Get the release and apply data persist logic defined on data_consumer function
# Verify that the processed data matches the provided release info (e.g. number of records)
def __stream_data_and_consume(release_info: ReleaseFileInfo, data_consumer: Callable[[dict], None]):
download_url = api_host + '/bulk/download/' + release_info.download_access_key
headers = {'Accept-Encoding': 'gzip', 'Authorization': 'Bearer ' + api_token}
record_counter = 0
first_lens_id = ''
last_lens_id = ''
with requests.get(download_url, headers=headers, stream=True) as r:
if r.status_code == requests.codes.too_many_requests:
raise RuntimeError('Download is rate limited. Please check your usage.')
with gzip.GzipFile(fileobj=r.raw) as f:
for line in f:
record = json.loads(line)
data_consumer(record)
record_counter += 1
if first_lens_id == '':
first_lens_id = record['lens_id']
last_lens_id = record['lens_id']
assert record_counter == release_info.num_records
assert first_lens_id == release_info.first_record_id
assert last_lens_id == release_info.last_record_id
# For snapshot consumer, insert the records to desired store.
def __consume_snapshot(record_json):
print('insert record with lens_id %s' % (record_json['lens_id']))
# For delta consumer, records can be either inserted, updated or deleted.
def __consume_delta(record_json):
if 'status' not in record_json:
raise RuntimeError('Incorrect delta file.')
lens_id = record_json['lens_id']
status = record_json['status']
record = record_json['data']
if status == 'UPDATED':
# Replace the record by lens_id
print('replace %s' % lens_id)
elif status == 'DELETED':
# Delete the record by lens_id. record_json['data'] will be null for DELETED
print('delete %s' % lens_id)
elif status == 'CREATED':
# Insert the record
print('insert %s' % lens_id)
else:
raise RuntimeError('Cannot find valid status (%s) for record %s' % (status, lens_id))
# Step 1: Process all available releases applying the delta sequentially
# Get all the data releases available
releases = __get_all_releases('patent')
print('Processing snapshot version %s and %d delta releases...' % (releases.snapshot.data_version, len(releases.delta)))
# Now apply the snapshot and persist the status into your tracker
if not __download_already_exists(releases.snapshot):
__stream_data_and_consume(releases.snapshot, __consume_snapshot)
__persist_latest_parse_info(releases.snapshot)
# Consume all subsequent delta if not already processed and track the progress
for delta in releases.delta:
if not __download_already_exists(delta):
__stream_data_and_consume(delta, __consume_delta)
__persist_latest_parse_info(delta)
# Step 2: After an initial run, its ideal to schedule run for new delta instead of checking/comparing all past releases.
new_release = __get_latest_release('patent', True)
if not __download_already_exists(new_release):
__stream_data_and_consume(new_release, __consume_delta)
__persist_latest_parse_info(releases.snapshot)
Download using UI
Bulk data files can also be downloaded manually from the Subscriptions tab in your Lens account. Active
bulk data plans include details of the latest bulk data file and a Download Bulk Data button for downloading the latest bulk data file.
Usage Endpoint
Please Note: Bulk data downloads via the Download API endpoint and UI are rate-limited. The number of allowed and remaining download requests per file can be viewed from the subscription usage endpoints. When downloading bulk data, you will receive status code 429 if the number of download requests has been exceeded. The usage endpoints are:
https://api.lens.org/subscriptions/patent_bulk/usagehttps://api.lens.org/subscriptions/scholarly_bulk/usage
Example response:
[
{
"remaining": 10,
"allowed": 10,
"frequency": "1 MONTH",
"type": "RESOURCE",
"note": "The allowed quota is applicable to unique resource access.",
"resources": {
"patent-202421-a1399131-beb0-4566-a3af-1ee8c1df7275.jsonl.gz": {
"remaining": 10,
"allowed": 10,
"frequency": "1 MONTH",
"type": "RESOURCE"
},
"patent-202427-99514b0e-5d5c-49c1-9223-de6c014b8923.jsonl.gz": {
"remaining": 10,
"allowed": 10,
"frequency": "1 MONTH",
"type": "RESOURCE"
},
"patent-202428-dd4bf7c0-adde-40e6-b138-f62976af8321.jsonl.gz": {
"remaining": 10,
"allowed": 10,
"frequency": "1 MONTH",
"type": "RESOURCE"
},
"patent-202429-41f2f86f-ef8f-4e0d-9b9d-08eaaa815eb5.jsonl.gz": {
"remaining": 10,
"allowed": 10,
"frequency": "1 MONTH",
"type": "RESOURCE"
}
}
}
]