What Is a Data Catalog?
A data catalog is a tool that helps organizations store, identify, and manage their data in an orderly, organized way. This catalog can include metadata (additional information about data) that lets users easily access data, evaluate data quality, and use data in their decisions.
The main features of a data catalog include:
- Data identification and discovery: Searching across data and different sources
- Metadata management: Storing information such as data types, formats, structure, and data change history
- Accessibility and collaboration: Sharing data for easier cross-team collaboration
Arvancloud's S3 Data Catalog is a managed data catalog based on Apache Iceberg that stores data and its metadata directly inside Object Storage buckets. This service provides a standard Iceberg REST interface so you can connect to it from engines you already use (such as Spark and PyIceberg) and use your buckets like a database with a flexible structure.
The Arvancloud Data Catalog makes it easy to turn Object Storage buckets into a data warehouse for all kinds of analytical workloads, including log analysis, business intelligence, and data pipelines.
What Is Apache Iceberg?
Apache Iceberg is an open table format designed to manage large-scale analytical data stored in object storage.
Its key features are:
- ACID transactions: Reliable concurrent reads and writes with full data integrity.
- Optimized metadata: Avoids full table scans by using indexed metadata for faster queries.
- Schema evolution: Add, rename, and drop columns without rewriting data.
Iceberg is currently widely supported by engines such as Apache Spark, Trino, Snowflake, DuckDB, and ClickHouse.
Why Do You Need a Data Catalog?
Although Iceberg data and metadata are stored directly in object storage, the list of tables and the pointers to their current metadata must be managed centrally by a data catalog.
Likewise, by providing coordinated, consistent access, data catalogs let different query engines safely read from and write to the same tables without causing data conflicts or corruption.
Enabling the Arvancloud Data Catalog
To use the Arvancloud Data Catalog, go to Object Storage in the panel. Then click Data Catalog, and on this page click the Data Catalog button.
On the new page, enter the data catalog's details.
First, set a name for the catalog, then select its bucket. If you don't have a bucket yet, you can create a new bucket on the same page. Also, for the access keys section, you must have created at least one machine user access key. The machine user guide helps you with this.
Click Create to create the catalog, and you can configure your tools with the Catalog URI, Warehouse Name, and your access key.
You can now use the catalog with any tool that supports the open table format, such as:
- PySpark
- PyIceberg
- DuckDB
- Trino
Data Processing Tools
Among the most popular data processing tools you can choose for using the data catalog are PyIceberg and PySpark.
Using PyIceberg to Connect to the Data Catalog
Before you start, you need to install the following libraries:
-
s3fs
-
PyIceberg
To install it, use the following command:
pip install pyiceberg -
PyArrow
To install it, use the following command:
pip install pyarrow
PyIceberg Sample Code
To connect to the data catalog and create Iceberg tables, you can use the following sample code:
import pyarrow as pa
from pyiceberg.catalog.rest import RestCatalog
from pyiceberg.exceptions import NamespaceAlreadyExistsError, TableAlreadyExistsError
import string
import random
# Define catalog connection details
WAREHOUSE = "<WAREHOUSE_NAME>"
CATALOG_URI = "https://<WAREHOUSE_NAME>.bucketcatalog.arvanstorage.ir/iceberg/"
TOKEN = "apikey XXXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
ACCESS_KEY = "XXXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
SECRET_ACCESS_KEY = "XXXXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
# This configuration can be adjusted based on your needs. For large batch requests,
# if you encounter timeouts, you can increase the timeout values.
S3_CONNECT_TIMEOUT = "30"
S3_REQUEST_TIMEOUT = "30"
# Connect to ArvanCloud Data Catalog
catalog = RestCatalog(
name="my_catalog",
uri= CATALOG_URI,
warehouse= WAREHOUSE,
token=TOKEN,
**{
"s3.connect-timeout": S3_CONNECT_TIMEOUT,
"s3.request-timeout": S3_REQUEST_TIMEOUT,
"type": "rest",
"s3.access-key-id": ACCESS_KEY,
"s3.secret-access-key": SECRET_ACCESS_KEY
}
)
# Create arvan namespace if it doesn't exist
try:
catalog.create_namespace("arvan")
except NamespaceAlreadyExistsError:
pass
# Define table schema
schema = pa.schema([
("id", pa.int64()),
("name", pa.string())
])
# Create empty Iceberg table
test_table = ("arvan", "my_table")
try:
table = catalog.create_table(
test_table,
schema=schema,
)
except TableAlreadyExistsError:
table = catalog.load_table(test_table)
# Create PyArrow table with data
data = pa.table({
"id": [1, 2, 3, 4, 5],
"name": ["Rostam", "Sohrab", "Esfandiar", "Kaykhosrow", "Siyavash"],
})
# Write data to the table
with table.transaction() as txn:
txn.append(data)
# Load the existing Iceberg table
table = catalog.load_table(test_table)
# Read data from table with limit
result = table.scan(limit=10).to_arrow() # Removed row_filter=None
# Print results
print(result)
Using PySpark to Connect to the Data Catalog
Before you start, you need to install the following:
-
Java 11
-
PySpark
To install it, use the following command:
pip install pyspark==3.5.0
PySpark Sample Code
To connect to the data catalog and run queries, you can use the following sample code:
from pyspark.sql import SparkSession
import uuid
import time
import random
import string
import time
# Define catalog connection details
WAREHOUSE = "<WAREHOUSE_NAME>"
CATALOG_URI = "https://<WAREHOUSE_NAME>.bucketcatalog.arvanstorage.ir/iceberg/"
TOKEN = "apikey XXXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
# Initialize Spark session with Iceberg configurations
spark = SparkSession.builder \
.appName("IcebergDataCatalogExample") \
.config('spark.jars.packages',
'org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.6.1,'
'org.apache.iceberg:iceberg-aws-bundle:1.6.1') \
.config("spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
.config("spark.sql.catalog.my_catalog", "org.apache.iceberg.spark.SparkCatalog") \
.config("spark.sql.catalog.my_catalog.type", "rest") \
.config("spark.sql.catalog.my_catalog.uri", CATALOG_URI) \
.config("spark.sql.catalog.my_catalog.warehouse", WAREHOUSE) \
.config("spark.sql.catalog.my_catalog.token", TOKEN) \
.getOrCreate()
try:
# Step 1: Use catalog and create schema
print("Step 1: Creating schema")
spark.sql("USE my_catalog")
spark.sql(f"CREATE NAMESPACE IF NOT EXISTS default")
print("Available schemas:")
schemas = spark.sql("SHOW NAMESPACES IN my_catalog").collect()
for row in schemas:
print(f" - {row['namespace']}")
# Step 2: Create table
print("\nStep 2: Creating table")
spark.sql(f"""
CREATE TABLE IF NOT EXISTS default.my_table (
id BIGINT,
name STRING
)
USING iceberg
""")
tables = spark.sql(f"SHOW TABLES IN default").collect()
print("Tables in schema:")
for row in tables:
print(f" - {row['tableName']}")
# Create a simple DataFrame
df = spark.createDataFrame(
[(1, "Rostam"), (2, "Sohrab"), (3, "Esfandiar"),(4, "Kaykhosrow"),(5, "Siyavash"),(6, "Faramarz")],
["id", "name"]
)
# Write the DataFrame to the Iceberg table
df.write \
.format("iceberg") \
.mode("append") \
.save("default.my_table")
# Read the data back from the Iceberg table
result_df = spark.read \
.format("iceberg") \
.load("default.my_table")
data_df = result_df.limit(5)
# Show the data
data = data_df.collect() # Collect the data from your DataFrame
for row in data:
# Print each row's data into the file
print(f"ID: {row['id']}, Name: {row['name']}\n")
except Exception as e:
print(f"Error occurred: {str(e)}")