通过 Iceberg REST Catalog 访问 Databricks Iceberg 表

云器 Lakehouse 通过 Iceberg REST Catalog 协议访问 Databricks Unity Catalog 中的 Iceberg 格式表，元数据实时同步，数据留在原 S3 存储不做搬迁。

预览

前置条件

• Databricks 工作区：支持 Unity Catalog
• 云器 Lakehouse 实例：与 Databricks 数据存储（S3）在同一云平台（均为 AWS）
• Iceberg 表数据存储在用户自己控制的 S3 bucket 中，并为 Lakehouse 配置跨账号读取权限
• Databricks Service Principal：已获取 OAuth Client ID 和 Secret

Databricks 侧准备

1. 获取 OAuth 凭据

https://accounts.cloud.databricks.com

2. 授予 Catalog 权限

GRANT USE CATALOG ON CATALOG <catalog_name> TO `<application-id>`; GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<application-id>`; GRANT SELECT ON SCHEMA <catalog_name>.<schema_name> TO `<application-id>`; GRANT EXTERNAL USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<application-id>`;

3. 开启 Metastore External Data Access

Databricks Workspace → Catalog → 齿轮图标 → Metastore → External data access → 打开。

4. Iceberg 表格式要求（关键）

SHOW TABLES

SELECT

table not found

在 Databricks 中建 Iceberg 表：

CREATE TABLE catalog.schema.my_iceberg_table USING ICEBERG LOCATION 's3://your-bucket/path/';

5. S3 存储要求

Iceberg 表数据必须在用户控制的 S3 bucket 中。Databricks 托管存储（

s3://dbstorage-prod-*

S3 403 Forbidden

同时 Lakehouse 的 AWS 账号需要对 S3 bucket 具有读权限。Lakehouse 的 AWS 账号 ID 可通过技术支持获取。

创建 Catalog Connection

CREATE CATALOG CONNECTION IF NOT EXISTS databricks_iceberg_conn TYPE ICEBERG_REST URI = 'https://<workspace>.cloud.databricks.com/api/2.1/unity-catalog/iceberg-rest' OAUTH_SERVER_URI = 'https://<workspace>.cloud.databricks.com/oidc/v1/token' ACCESS_REGION = '<s3-bucket-region>' OAUTH_CLIENT_ID = '<oauth-client-id>' OAUTH_CLIENT_SECRET = '<oauth-client-secret>' OAUTH_SCOPE = 'all-apis' WAREHOUSE = '<databricks-catalog-name>';

参数说明

TYPE

ICEBERG_REST

=

URI

/iceberg-rest

/iceberg

OAUTH_SERVER_URI

Credential was not sent

ACCESS_REGION

OAUTH_CLIENT_ID

OAUTH_CLIENT_SECRET

OAUTH_SCOPE

all-apis

WAREHOUSE

workspace

main

TYPE = ICEBERG_REST

=

TYPE ICEBERG_REST

创建 External Catalog

CREATE EXTERNAL CATALOG databricks_iceberg_catalog CONNECTION databricks_iceberg_conn;

OPTIONS ('catalog' = '...')

验证

-- 验证连通性 SHOW SCHEMAS IN databricks_iceberg_catalog; -- 查看表列表 SHOW TABLES IN databricks_iceberg_catalog.<schema>; -- 查看表结构 DESC TABLE databricks_iceberg_catalog.<schema>.<iceberg_table>; -- 查询数据 SELECT * FROM databricks_iceberg_catalog.<schema>.<iceberg_table> LIMIT 10;

常见错误

Credential was not sent or was of an unsupported type

Credential was not sent or was of an unsupported type

OAUTH_SERVER_URI

Legacy Iceberg endpoints deprecated

URI

/iceberg

/iceberg-rest

Must provide 'warehouse' parameter

WAREHOUSE

S3 403 Forbidden

Iceberg 表数据存在 Databricks 托管存储（

s3://dbstorage-prod-*

table or view not found（SELECT 时）

该表不是 Iceberg 格式（可能是 Delta / Parquet / CSV）。SHOW TABLES 能显示所有格式，但只有 Iceberg 格式能通过 SELECT 查询。

管理

-- 查看连接详情 DESC CONNECTION databricks_iceberg_conn; -- 删除 Catalog DROP CATALOG databricks_iceberg_catalog; -- 删除连接 DROP CONNECTION databricks_iceberg_conn;

相关文档

• 创建 Catalog Connection — 完整 DDL 语法
• Databricks Unity Catalog 联邦查询实践 — TYPE DATABRICKS 方式
• External Catalog 联邦查询