How to Convert String to Integer in PySpark DataFrame?

To convert a string column to an integer in a PySpark DataFrame, you can use the `withColumn` function along with the `cast` method. Below is a detailed step-by-step explanation along with a code snippet demonstrating this process.

Step-by-Step Explanation

1. **Initialize Spark Session**: Start by initializing a Spark session.
2. **Create DataFrame**: Create a DataFrame that contains a string column.
3. **Convert String to Integer**: Use the `withColumn` function and `cast` method to transform the string column to an integer column.
4. **Show the Results**: Display the results to verify the conversion.

Code Snippet

Below is a PySpark example that demonstrates converting a string column to an integer column:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

# Step 1: Initialize Spark Session
spark = SparkSession.builder \
    .appName("StringToIntConversion") \
    .getOrCreate()

# Step 2: Create DataFrame with a string column
data = [("1", "Alice"), ("2", "Bob"), ("3", "Cathy")]
columns = ["id", "name"]
df = spark.createDataFrame(data, columns)

# Step 3: Convert String to Integer using withColumn and cast method
df = df.withColumn("id", col("id").cast("int"))

# Step 4: Show the results
df.show()

Output

+---+-----+
| id| name|
+---+-----+
|  1|Alice|
|  2|  Bob|
|  3|Cathy|
+---+-----+

In this example, the `id` column was initially of type string. We used the `withColumn` function along with the `cast` method to change its type to integer. The `df.show()` function is then used to display the DataFrame, and you can see that the `id` column now contains integers.