mirror of
https://github.com/grafana/mimir.git
synced 2026-03-21 14:51:10 +00:00
Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
ac515c3689#### What this PR does
The compactor scheduler persists using a bbolt database, but bbolt only
supports a single write at a time. This introduces sharding tenants
across multiple bbolt databases in the scheduler to improve write
throughput. The configuration that used to be the database path has been
changed to a directory where these shards will be located.
A consistent hash is used to assign tenants to shards. Changing the
number of shards through configuration is supported and tenant data is
migrated between the shards on startup if the shards changed. The
stability of the hash is not a requirement because `RecoverAll` binds
tenants to a `JobPersister` through a full scan. `Drop()` was added to
`JobPersister` to never have to rediscover the associated shard.
The migration procedure upon shard change was the most complex part of
this change. To ensure crash safety the operations are ordered to
perform copies before any deletions. The modifications to each database
are batched at each stage. In order to know when a migration procedure
was completed (and to detect if a shard disappeared later on) a metadata
object is written to the first shard to persist the intended number of
shards.
As a note, I separated bbolt specific logic from `persistence.go` into
`persistence_bbolt.go` in order to make the interfaces easier to see.
#### Checklist
- [x] Tests updated.
- [ ] Documentation added.
- [ ] `CHANGELOG.md` updated - the order of entries should be
`[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry
is not needed, please add the `changelog-not-needed` label to the PR.
- [ ]
[`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md)
updated with experimental features.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Introduces multi-file bbolt sharding and on-startup migration logic,
which impacts durability and recovery paths; failures could lead to lost
or duplicated persisted jobs if edge cases exist.
>
> **Overview**
> The scheduler’s bbolt persistence is reworked to **shard tenants
across multiple bbolt database files** (directory-based storage) to
improve write throughput, using consistent hashing to select the shard
per tenant.
>
> Startup now **prepares/migrates shards** when `shard_count` changes by
copying tenant buckets to their new shard, deleting old placements, and
persisting shard layout in a new `PersistenceMetadata` proto stored in a
reserved metadata bucket.
>
> Configuration is updated from `BboltPath` to a structured
`BboltConfig` (`dir`, `shard_count`), `JobPersister` gains `Drop()` for
shard-aware tenant deletion, and tests are moved/expanded to cover
sharding and scale up/down migration behavior.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
|
||
|---|---|---|
| .config/go | ||
| .github | ||
| cmd | ||
| development | ||
| docs | ||
| images | ||
| integration | ||
| mimir-build-image | ||
| operations | ||
| packaging | ||
| pkg | ||
| tools | ||
| vendor | ||
| .gitattributes | ||
| .gitconfig | ||
| .gitignore | ||
| .golangci.yml | ||
| .lintignore | ||
| .prettierignore | ||
| ADOPTERS.md | ||
| AGENTS.md | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| CODE_OF_CONDUCT.md | ||
| CODEOWNERS | ||
| CONTRIBUTING.md | ||
| go.mod | ||
| go.sum | ||
| GOVERNANCE.md | ||
| LICENSE | ||
| LICENSING.md | ||
| MAINTAINERS.md | ||
| Makefile | ||
| Makefile.local.example | ||
| README.md | ||
| RELEASE.md | ||
| renovate.json5 | ||
| VERSION | ||
Grafana Mimir
Grafana Mimir is an open source software project that provides a scalable long-term storage for Prometheus. Some of the core strengths of Grafana Mimir include:
- Easy to install and maintain: Grafana Mimir’s extensive documentation, tutorials, and deployment tooling make it quick to get started. Using its monolithic mode, you can get Grafana Mimir up and running with just one binary and no additional dependencies. Once deployed, the best-practice dashboards, alerts, and runbooks packaged with Grafana Mimir make it easy to monitor the health of the system.
- Massive scalability: You can run Grafana Mimir's horizontally-scalable architecture across multiple machines, resulting in the ability to process orders of magnitude more time series than a single Prometheus instance. Internal testing shows that Grafana Mimir handles up to 1 billion active time series.
- Global view of metrics: Grafana Mimir enables you to run queries that aggregate series from multiple Prometheus instances, giving you a global view of your systems. Its query engine extensively parallelizes query execution, so that even the highest-cardinality queries complete with blazing speed.
- Cheap, durable metric storage: Grafana Mimir uses object storage for long-term data storage, allowing it to take advantage of this ubiquitous, cost-effective, high-durability technology. It is compatible with multiple object store implementations, including AWS S3, Google Cloud Storage, Azure Blob Storage, OpenStack Swift, as well as any S3-compatible object storage.
- High availability: Grafana Mimir replicates incoming metrics, ensuring that no data is lost in the event of machine failure. Its horizontally scalable architecture also means that it can be restarted, upgraded, or downgraded with zero downtime, which means no interruptions to metrics ingestion or querying.
- Natively multi-tenant: Grafana Mimir’s multi-tenant architecture enables you to isolate data and queries from independent teams or business units, making it possible for these groups to share the same cluster. Advanced limits and quality-of-service controls ensure that capacity is shared fairly among tenants.
Migrating to Grafana Mimir
If you're migrating to Grafana Mimir, refer to the following documents:
Deploying Grafana Mimir
For information about how to deploy Grafana Mimir, refer to Deploy Grafana Mimir.
Getting started
If you’re new to Grafana Mimir, read the Get started guide.
Before deploying Grafana Mimir in a production environment, read:
Documentation
Refer to the following links to access Grafana Mimir documentation:
- Latest release
- Upcoming release, at the tip of the
mainbranch
Contributing
To contribute to Grafana Mimir, refer to Contributing to Grafana Mimir.
Join the Grafana Mimir discussion
If you have any questions or feedback regarding Grafana Mimir, join the Grafana Mimir Discussion. Alternatively, consider joining the monthly Grafana Mimir Community Call.
Your feedback is always welcome, and you can also share it via the #mimir Slack channel.
License
Grafana Mimir is distributed under AGPL-3.0-only.