How Do You Change DataFrame Column Names in PySpark?

In PySpark, changing DataFrame column names can be achieved using various methods. I’ll explain some of the common methods for renaming columns with examples.

Using the `withColumnRenamed` Method

The `withColumnRenamed` method is used to rename a specific column. It’s useful when you only need to rename a single column.

Example:

from pyspark.sql import SparkSession

# Initialize SparkSession
spark = SparkSession.builder \
    .appName("Rename Column Example") \
    .getOrCreate()

# Sample DataFrame
data = [("James", "Smith", "USA"), ("Michael", "Rose", "USA"), ("Robert", "Williams", "USA")]
columns = ["Firstname", "Lastname", "Country"]

df = spark.createDataFrame(data, schema=columns)

# Rename 'Firstname' to 'First_Name'
df_renamed = df.withColumnRenamed("Firstname", "First_Name")
df_renamed.show()

Output:

+----------+--------+-------+
|First_Name|Lastname|Country|
+----------+--------+-------+
|     James|   Smith|    USA|
|   Michael|    Rose|    USA|
|    Robert|Williams|    USA|
+----------+--------+-------+

Using the `toDF` Method

The `toDF` method can be used to rename all columns in the DataFrame. This method is useful when you need to change multiple column names at once.

Example:

# Rename all columns using toDF
new_columns = ["First_Name", "Last_Name", "Country_Name"]
df_renamed_all = df.toDF(*new_columns)
df_renamed_all.show()

Output:

+----------+---------+------------+
|First_Name|Last_Name|Country_Name|
+----------+---------+------------+
|     James|    Smith|         USA|
|   Michael|     Rose|         USA|
|    Robert| Williams|         USA|
+----------+---------+------------+

Using the `alias` Method in `selectExpr`

Another method to rename multiple columns is using the `selectExpr` with aliasing.

Example:

# Rename columns using selectExpr and alias method
df_alias = df.selectExpr("Firstname as First_Name", "Lastname as Last_Name", "Country as Country_Name")
df_alias.show()

Output:

+----------+---------+------------+
|First_Name|Last_Name|Country_Name|
+----------+---------+------------+
|     James|    Smith|         USA|
|   Michael|     Rose|         USA|
|    Robert| Williams|         USA|
+----------+---------+------------+

Comparison Table of Methods

Here’s a comparison table of the different methods to rename columns in PySpark:

Each method has its own advantages depending on the situation and requirement. Choose the method that best fits your use case.