我已经构建了一个具有复制数据活动的管道,该活动从Azure Data Lake
并将其输出到Azure Blob Storage
.
在输出中,我可以看到我的某些行没有数据,我想将它们从副本中排除。在以下示例中,第二行没有有用的数据:
{"TenantId":"qa","Timestamp":"2019-03-06T10:53:51.634Z","PrincipalId":2,"ControlId":"729c3b6e-0442-4884-936c-c36c9b466e9d","ZoneInternalId":0,"IsAuthorized":true,"PrincipalName":"John","StreetName":"Rue 1","ExemptionId":8}
{"TenantId":"qa","Timestamp":"2019-03-06T10:59:09.74Z","PrincipalId":null,"ControlId":null,"ZoneInternalId":null,"IsAuthorized":null,"PrincipalName":null,"StreetName":null,"ExemptionId":null}
Question
在“复制数据”活动中,如何设置规则来排除缺少某些值的行?
这是我的管道的代码:
{
"name": "pipeline1",
"properties": {
"activities": [
{
"name": "Copy from Data Lake to Blob",
"type": "Copy",
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [
{
"name": "Source",
"value": "tenantdata/events/"
},
{
"name": "Destination",
"value": "controls/"
}
],
"typeProperties": {
"source": {
"type": "AzureDataLakeStoreSource",
"recursive": true
},
"sink": {
"type": "BlobSink",
"copyBehavior": "MergeFiles"
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"columnMappings": {
"Body.TenantId": "TenantId",
"Timestamp": "Timestamp",
"Body.PrincipalId": "PrincipalId",
"Body.ControlId": "ControlId",
"Body.ZoneId": "ZoneInternalId",
"Body.IsAuthorized": "IsAuthorized",
"Body.PrincipalName": "PrincipalName",
"Body.StreetName": "StreetName",
"Body.Exemption.Kind": "ExemptionId"
}
}
},
"inputs": [
{
"referenceName": "qadl",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "datalakestaging",
"type": "DatasetReference"
}
]
}
]
}
}