Buckets:
| # @keyv/bigmap [<img width="100" align="right" src="https://jaredwray.com/images/keyv-symbol.svg" alt="keyv">](https://github.com/jaredwra/keyv) | |
| > Bigmap for Keyv | |
| [](https://github.com/jaredwray/keyv/actions/workflows/tests.yaml) | |
| [](https://codecov.io/gh/jaredwray/keyv) | |
| [](https://www.npmjs.com/package/@keyv/bigmap) | |
| [](https://npmjs.com/package/@keyv/bigmap) | |
| # Features | |
| * Based on the Map interface and uses the same API. | |
| * Lightweight with no dependencies. | |
| * Scales to past the 17 million key limit of a regular Map. | |
| * Uses a hash `djb2Hash` for fast key lookups. | |
| * Ability to use your own hash function. | |
| * Built in Typescript and Generics for type safety. | |
| * Used in `@cacheable/memory` for scalable in-memory caching. | |
| * Maintained regularly with a focus on performance and reliability. | |
| # Table of Contents | |
| - [Features](#features) | |
| - [Installation](#installation) | |
| - [Overview](#overview) | |
| - [Basic Usage](#basic-usage) | |
| - [Custom Store Size](#custom-store-size) | |
| - [Custom Hash Function](#custom-hash-function) | |
| - [Iteration](#iteration) | |
| - [For...of Loop](#forof-loop) | |
| - [forEach](#foreach) | |
| - [Keys, Values, and Entries](#keys-values-and-entries) | |
| - [Advanced Features](#advanced-features) | |
| - [Type Safety with Generics](#type-safety-with-generics) | |
| - [Large-Scale Data](#large-scale-data) | |
| - [Using with Keyv](#using-with-keyv) | |
| - [createKeyv](#createkeyv) | |
| - [With Custom Options](#with-custom-options) | |
| - [Type Safety](#type-safety) | |
| - [Integration with Keyv Ecosystem](#integration-with-keyv-ecosystem) | |
| - [API](#api) | |
| - [Constructor](#constructor) | |
| - [Properties](#properties) | |
| - [Methods](#methods) | |
| - [set](#set) | |
| - [get](#get) | |
| - [has](#has) | |
| - [delete](#delete) | |
| - [clear](#clear) | |
| - [forEach](#foreach) | |
| - [keys](#keys) | |
| - [values](#values) | |
| - [entries](#entries) | |
| - [Symbol.iterator](#symboliterator) | |
| - [getStore](#getstorekey) | |
| - [getStoreMap](#getstoremapindex) | |
| - [initStore](#initstore) | |
| - [Types](#types) | |
| - [StoreHashFunction](#storehashfunction) | |
| - [defaultHashFunction(key, storeSize)](#defaulthashfunctionkey-storesize) | |
| - [Contributing](#contributing) | |
| - [License](#license) | |
| # Installation | |
| ```bash | |
| npm install --save keyv @keyv/bigmap | |
| ``` | |
| # Overview | |
| BigMap is a scalable Map implementation that overcomes JavaScript's built-in Map limit of approximately 17 million entries. It uses a distributed hash approach with multiple internal Map instances. | |
| # Basic Usage | |
| ```typescript | |
| import { BigMap } from '@keyv/bigmap'; | |
| // Create a new BigMap | |
| const bigMap = new BigMap<string, number>(); | |
| // Set values | |
| bigMap.set('key1', 100); | |
| bigMap.set('key2', 200); | |
| // Get values | |
| const value = bigMap.get('key1'); // 100 | |
| // Check if key exists | |
| bigMap.has('key1'); // true | |
| // Delete a key | |
| bigMap.delete('key1'); // true | |
| // Get size | |
| console.log(bigMap.size); // 1 | |
| // Clear all entries | |
| bigMap.clear(); | |
| ``` | |
| # Custom Store Size | |
| By default, BigMap uses 4 internal Map instances. You can configure this: | |
| ```typescript | |
| const bigMap = new BigMap<string, number>({ storeSize: 10 }); | |
| ``` | |
| **Note:** Changing the `storeSize` after initialization will clear all entries. | |
| # Custom Hash Function | |
| Provide your own hash function for key distribution: | |
| ```typescript | |
| const customHashFunction = (key: string, storeSize: number) => { | |
| return key.length % storeSize; | |
| }; | |
| const bigMap = new BigMap<string, string>({ | |
| storeHashFunction: customHashFunction | |
| }); | |
| ``` | |
| ## Using Hashery for Hash Functions | |
| [Hashery](https://github.com/jaredwray/hashery) is a powerful hashing library that provides multiple hash algorithms. You can use it for better key distribution and it is available as an export: | |
| ```typescript | |
| import { BigMap, Hashery } from '@keyv/bigmap'; | |
| const hashery = new Hashery(); | |
| // Using Hashery's toNumberSync for deterministic key distribution | |
| const bigMap = new BigMap<string, string>({ | |
| storeHashFunction: (key: string, storeSize: number) => { | |
| return hashery.toNumberSync(key, { min: 0, max: storeSize - 1 }); | |
| } | |
| }); | |
| // You can also use different algorithms | |
| const hasheryFnv1 = new Hashery({ defaultAlgorithmSync: 'fnv1' }); | |
| const bigMapWithFnv1 = new BigMap<string, string>({ | |
| storeHashFunction: (key: string, storeSize: number) => { | |
| return hasheryFnv1.toNumberSync(key, { min: 0, max: storeSize - 1 }); | |
| } | |
| }); | |
| ``` | |
| Hashery supports multiple synchronous hash algorithms: | |
| - **djb2** - Fast hash function (default) | |
| - **fnv1** - Excellent distribution for hash tables | |
| - **murmer** - MurmurHash algorithm | |
| - **crc32** - Cyclic Redundancy Check | |
| # Iteration | |
| BigMap supports all standard Map iteration methods: | |
| ## For...of Loop | |
| ```typescript | |
| const bigMap = new BigMap<string, number>(); | |
| bigMap.set('a', 1); | |
| bigMap.set('b', 2); | |
| for (const [key, value] of bigMap) { | |
| console.log(key, value); | |
| } | |
| ``` | |
| ## forEach | |
| ```typescript | |
| bigMap.forEach((value, key) => { | |
| console.log(key, value); | |
| }); | |
| // With custom context | |
| const context = { sum: 0 }; | |
| bigMap.forEach(function(value) { | |
| this.sum += value; | |
| }, context); | |
| ``` | |
| ## Keys, Values, and Entries | |
| ```typescript | |
| // Iterate over keys | |
| for (const key of bigMap.keys()) { | |
| console.log(key); | |
| } | |
| // Iterate over values | |
| for (const value of bigMap.values()) { | |
| console.log(value); | |
| } | |
| // Iterate over entries | |
| for (const [key, value] of bigMap.entries()) { | |
| console.log(key, value); | |
| } | |
| ``` | |
| # Advanced Features | |
| ## Type Safety with Generics | |
| ```typescript | |
| interface User { | |
| id: number; | |
| name: string; | |
| } | |
| const userMap = new BigMap<string, User>(); | |
| userMap.set('user1', { id: 1, name: 'Alice' }); | |
| ``` | |
| ## Large-Scale Data | |
| BigMap is designed to handle millions of entries: | |
| ```typescript | |
| const bigMap = new BigMap<string, number>({ storeSize: 16 }); | |
| // Add 20+ million entries without hitting Map limits | |
| for (let i = 0; i < 20000000; i++) { | |
| bigMap.set(`key${i}`, i); | |
| } | |
| console.log(bigMap.size); // 20000000 | |
| ``` | |
| # Using with Keyv | |
| BigMap can be used as a storage adapter for [Keyv](https://github.com/jaredwray/keyv), providing a scalable in-memory store with TTL support. | |
| ## createKeyv | |
| The `createKeyv` function creates a Keyv instance with BigMap as the storage adapter. | |
| **Parameters:** | |
| - `options` (optional): BigMap configuration options | |
| - `storeSize` (number): Number of internal Map instances. Default: `4` | |
| - `storeHashFunction` (StoreHashFunction): Custom hash function for key distribution | |
| **Returns:** `Keyv` instance with BigMap adapter | |
| **Example:** | |
| ```typescript | |
| import { createKeyv } from '@keyv/bigmap'; | |
| // Basic usage | |
| const keyv = createKeyv(); | |
| // Set with TTL (in milliseconds) | |
| await keyv.set('user:123', { name: 'Alice', age: 30 }, 60000); // Expires in 60 seconds | |
| // Get value | |
| const user = await keyv.get('user:123'); | |
| console.log(user); // { name: 'Alice', age: 30 } | |
| // Check if key exists | |
| const exists = await keyv.has('user:123'); | |
| // Delete key | |
| await keyv.delete('user:123'); | |
| // Clear all keys | |
| await keyv.clear(); | |
| ``` | |
| ## With Custom Options | |
| ```typescript | |
| import { createKeyv } from '@keyv/bigmap'; | |
| // Create with custom store size for better performance with millions of keys | |
| const keyv = createKeyv({ storeSize: 16 }); | |
| // With custom hash function | |
| const keyv = createKeyv({ | |
| storeSize: 8, | |
| storeHashFunction: (key, storeSize) => { | |
| // Custom distribution logic | |
| return key.length % storeSize; | |
| } | |
| }); | |
| ``` | |
| ## Type Safety | |
| ```typescript | |
| import { createKeyv } from '@keyv/bigmap'; | |
| interface Product { | |
| id: string; | |
| name: string; | |
| price: number; | |
| } | |
| const keyv = createKeyv<string, Product>(); | |
| await keyv.set('product:1', { | |
| id: '1', | |
| name: 'Laptop', | |
| price: 999 | |
| }); | |
| const product = await keyv.get<Product>('product:1'); | |
| ``` | |
| # Integration with Keyv Ecosystem | |
| BigMap works seamlessly with the Keyv ecosystem: | |
| ```typescript | |
| import { createKeyv } from '@keyv/bigmap'; | |
| const cache = createKeyv({ storeSize: 16 }); | |
| // Use with namespaces | |
| const users = cache.namespace('users'); | |
| const products = cache.namespace('products'); | |
| await users.set('123', { name: 'Alice' }); | |
| await products.set('456', { name: 'Laptop' }); | |
| // Iterate over keys | |
| for await (const [key, value] of cache.iterator()) { | |
| console.log(key, value); | |
| } | |
| ``` | |
| # API | |
| ## Constructor | |
| `new BigMap<K, V>(options?)` | |
| Creates a new BigMap instance. | |
| **Parameters:** | |
| - `options` (optional): Configuration options | |
| - `storeSize` (number): Number of internal Map instances to use. Default: `4`. Must be at least 1. | |
| - `storeHashFunction` (StoreHashFunction): Custom hash function for key distribution. Default: `defaultHashFunction` | |
| **Example:** | |
| ```typescript | |
| const bigMap = new BigMap<string, number>(); | |
| const customBigMap = new BigMap<string, number>({ | |
| storeSize: 10, | |
| storeHashFunction: (key, storeSize) => key.length % storeSize | |
| }); | |
| ``` | |
| ## Properties | |
| | Property | Type | Access | Description | | |
| |----------|------|--------|-------------| | |
| | `size` | `number` | Read-only | Gets the total number of entries in the BigMap. | | |
| | `storeSize` | `number` | Read/Write | Gets or sets the number of internal Map instances. **Note:** Setting this will clear all entries. Default: `4` | | |
| | `storeHashFunction` | `StoreHashFunction \| undefined` | Read/Write | Gets or sets the hash function used for key distribution. | | |
| | `store` | `Array<Map<K, V>>` | Read-only | Gets the internal array of Map instances. | | |
| **Examples:** | |
| ```typescript | |
| const bigMap = new BigMap<string, number>(); | |
| // size property | |
| bigMap.set('key1', 100); | |
| console.log(bigMap.size); // 1 | |
| // storeSize property | |
| console.log(bigMap.storeSize); // 4 (default) | |
| bigMap.storeSize = 8; // Changes size and clears all entries | |
| // storeHashFunction property | |
| bigMap.storeHashFunction = (key, storeSize) => key.length % storeSize; | |
| // store property | |
| console.log(bigMap.store.length); // 8 | |
| ``` | |
| ## Methods | |
| ### set | |
| Sets the value for a key in the map. | |
| **Parameters:** | |
| - `key` (K): The key to set | |
| - `value` (V): The value to associate with the key | |
| **Returns:** `Map<K, V>` - The internal Map instance where the key was stored | |
| **Example:** | |
| ```typescript | |
| bigMap.set('user123', { name: 'Alice' }); | |
| ``` | |
| ### get | |
| Gets the value associated with a key. | |
| **Parameters:** | |
| - `key` (K): The key to retrieve | |
| **Returns:** `V | undefined` - The value, or undefined if not found | |
| **Example:** | |
| ```typescript | |
| const value = bigMap.get('user123'); | |
| ``` | |
| ### has | |
| Checks if a key exists in the map. | |
| **Parameters:** | |
| - `key` (K): The key to check | |
| **Returns:** `boolean` - True if the key exists, false otherwise | |
| **Example:** | |
| ```typescript | |
| if (bigMap.has('user123')) { | |
| console.log('User exists'); | |
| } | |
| ``` | |
| ### delete | |
| Deletes a key-value pair from the map. | |
| **Parameters:** | |
| - `key` (K): The key to delete | |
| **Returns:** `boolean` - True if the entry was deleted, false if the key was not found | |
| **Example:** | |
| ```typescript | |
| const deleted = bigMap.delete('user123'); | |
| ``` | |
| ### clear | |
| Removes all entries from the map. | |
| **Returns:** `void` | |
| **Example:** | |
| ```typescript | |
| bigMap.clear(); | |
| console.log(bigMap.size); // 0 | |
| ``` | |
| ### forEach | |
| Executes a provided function once for each key-value pair. | |
| **Parameters:** | |
| - `callbackfn` (function): Function to execute for each entry | |
| - `value` (V): The value of the current entry | |
| - `key` (K): The key of the current entry | |
| - `map` (`Map<K, V>`): The BigMap instance | |
| - `thisArg` (optional): Value to use as `this` when executing the callback | |
| **Returns:** `void` | |
| **Example:** | |
| ```typescript | |
| bigMap.forEach((value, key) => { | |
| console.log(`${key}: ${value}`); | |
| }); | |
| // With custom context | |
| const context = { total: 0 }; | |
| bigMap.forEach(function(value) { | |
| this.total += value; | |
| }, context); | |
| ``` | |
| ### keys | |
| Returns an iterator of all keys in the map. | |
| **Returns:** `IterableIterator<K>` | |
| **Example:** | |
| ```typescript | |
| for (const key of bigMap.keys()) { | |
| console.log(key); | |
| } | |
| ``` | |
| ### values | |
| Returns an iterator of all values in the map. | |
| **Returns:** `IterableIterator<V>` | |
| **Example:** | |
| ```typescript | |
| for (const value of bigMap.values()) { | |
| console.log(value); | |
| } | |
| ``` | |
| ### entries | |
| Returns an iterator of all key-value pairs in the map. | |
| **Returns:** `IterableIterator<[K, V]>` | |
| **Example:** | |
| ```typescript | |
| for (const [key, value] of bigMap.entries()) { | |
| console.log(key, value); | |
| } | |
| ``` | |
| ### Symbol.iterator | |
| Returns an iterator for the map (same as `entries()`). Enables `for...of` loops. | |
| **Returns:** `IterableIterator<[K, V]>` | |
| **Example:** | |
| ```typescript | |
| for (const [key, value] of bigMap) { | |
| console.log(key, value); | |
| } | |
| ``` | |
| ### getStore | |
| Gets the internal Map instance for a specific key. | |
| **Parameters:** | |
| - `key` (K): The key to find the store for | |
| **Returns:** `Map<K, V>` - The internal Map instance | |
| **Example:** | |
| ```typescript | |
| const store = bigMap.getStore('user123'); | |
| ``` | |
| ### getStoreMap | |
| Gets the internal Map instance at a specific index. | |
| **Parameters:** | |
| - `index` (number): The index of the Map to retrieve (0 to storeSize - 1) | |
| **Returns:** `Map<K, V>` - The Map at the specified index | |
| **Throws:** Error if index is out of bounds | |
| **Example:** | |
| ```typescript | |
| const firstMap = bigMap.getStoreMap(0); | |
| ``` | |
| ### initStore | |
| Initializes the internal store with empty Map instances. Called automatically during construction. | |
| **Returns:** `void` | |
| ## Types | |
| ### StoreHashFunction | |
| Type definition for custom hash functions. | |
| ```typescript | |
| type StoreHashFunction = (key: string, storeSize: number) => number; | |
| ``` | |
| **Parameters:** | |
| - `key` (string): The key to hash (converted to string) | |
| - `storeSize` (number): The number of stores (adjusted for zero-based index) | |
| **Returns:** `number` - The index of the store to use (0 to storeSize - 1) | |
| ### defaultHashFunction | |
| The default hash function using DJB2 algorithm from [Hashery](https://npmjs.com/package/hashery): | |
| **Example:** | |
| ```typescript | |
| import { defaultHashFunction } from '@keyv/bigmap'; | |
| const index = defaultHashFunction('myKey', 4); | |
| ``` | |
| ### djb2Hash | |
| DJB2 hash algorithm implementation. | |
| **Parameters:** | |
| - `string` (string): The string to hash | |
| - `min` (number): Minimum value. Default: `0` | |
| - `max` (number): Maximum value. Default: `10` | |
| **Returns:** `number` - Hash value within the specified range | |
| **Example:** | |
| ```typescript | |
| import { djb2Hash } from '@keyv/bigmap'; | |
| const hash = djb2Hash('myKey', 0, 10); | |
| ``` | |
| # Contributing | |
| Please see our [contributing](https://github.com/jaredwray/keyv/blob/main/CONTRIBUTING.md) guide. | |
| # License | |
| [MIT © Jared Wray](LICENSE) | |
Xet Storage Details
- Size:
- 14.5 kB
- Xet hash:
- 99f5ddbc6923d513d751bacc00b4d9754ad24b3964a8ad97f4d797d60f5ee332
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.