AWS Glue 到 Redshift：是否可以替换、更新或删除数据？

以下是我如何设置的一些要点：

• 我将 CSV 文件上传到 S3，并设置了 Glue 爬虫来创建表和架构。
• 我有一个 Glue 作业设置，它使用 JDBC 连接将数据从 Glue 表写入我们的 Amazon Redshift 数据库。该作业还负责映射列并创建红移表。

通过重新运行作业，我在 redshift 中得到了重复的行（如预期）。但是，有没有办法在插入新数据之前使用键或胶水中的分区设置来替换或删除行？

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

from awsglue.dynamicframe import DynamicFrame
from awsglue.transforms import SelectFields

from pyspark.sql.functions import lit

## @params: [TempDir, JOB_NAME]
args = getResolvedOptions(sys.argv, ['TempDir','JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

columnMapping = [
    ("id", "int", "id", "int"),
    ("name", "string", "name", "string"),
]

datasource1 = glueContext.create_dynamic_frame.from_catalog(database = "db01", table_name = "table01", transformation_ctx = "datasource0")

applymapping1 = ApplyMapping.apply(frame = datasource1, mappings = columnMapping, transformation_ctx = "applymapping1")
resolvechoice1 = ResolveChoice.apply(frame = applymapping1, choice = "make_cols", transformation_ctx = "resolvechoice1")
dropnullfields1 = DropNullFields.apply(frame = resolvechoice1, transformation_ctx = "dropnullfields1")
df1 = dropnullfields1.toDF()
data1 = df1.withColumn('platform', lit('test'))
data1 = DynamicFrame.fromDF(data1, glueContext, "data_tmp1")

## Write data to redshift
datasink1 = glueContext.write_dynamic_frame.from_jdbc_conf(frame = data1, catalog_connection = "Test Connection", connection_options = {"dbtable": "table01", "database": "db01"}, redshift_tmp_dir = args["TempDir"], transformation_ctx = "datasink1")

job.commit()

工作书签是关键。只需编辑作业并启用“作业书签”，它就不会处理已处理的数据。 请注意，作业必须重新运行一次才能检测到不必再次重新处理旧数据。

欲了解更多信息，请参阅：http://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html http://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html

在我看来，“书签”这个名字有点牵强。如果不是我在搜索过程中偶然发现的，我绝对不会看它。